Nov 03 20:28:02.588395 ip-10-0-128-49 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:28:03.066997 ip-10-0-128-49 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:03.066997 ip-10-0-128-49 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:28:03.066997 ip-10-0-128-49 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:03.066997 ip-10-0-128-49 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:03.072576 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072485    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:03.072576 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072574    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:03.072576 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072579    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:03.072576 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072582    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072585    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072589    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072592    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072595    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072598    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072600    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072603    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072606    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072609    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072611    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072614    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072616    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072619    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072621    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072624    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072627    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072629    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072632    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072634    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:03.072701 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072641    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072644    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072647    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072649    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072652    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072655    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072657    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072660    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072662    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072665    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072667    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072669    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072672    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072674    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072677    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072680    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072682    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072685    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072688    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072690    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:03.073198 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072693    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072695    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072698    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072700    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072702    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072705    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072710    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072715    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072718    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072721    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072724    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072727    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072730    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072733    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072736    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072738    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072742    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072744    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072747    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:03.073707 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072749    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072752    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072754    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072757    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072760    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072763    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072765    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072768    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072771    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072773    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072775    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072778    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072781    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072783    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072787    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072790    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072792    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072794    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072797    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:03.074179 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072799    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072802    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072804    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072806    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072810    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072812    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072815    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072817    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072820    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072823    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072826    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072828    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072830    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072840    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072844    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072849    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072853    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072855    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072858    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:03.074634 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072861    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072864    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072867    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072869    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072873    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.072876    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073739    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073747    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073750    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073752    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073755    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073758    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073761    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073764    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073766    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073769    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073772    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073774    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073777    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073779    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:03.075073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073782    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073784    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073788    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073791    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073794    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073796    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073799    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073802    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073804    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073807    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073809    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073812    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073814    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073819    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073822    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073825    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073827    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073830    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073833    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:03.075583 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073836    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073838    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073841    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073844    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073847    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073849    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073851    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073855    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073857    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073860    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073863    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073865    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073868    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073870    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073873    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073875    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073878    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073881    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073883    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073886    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:03.076040 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073888    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073890    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073893    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073895    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073898    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073900    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073905    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073907    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073910    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073912    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073916    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073921    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073928    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073931    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073936    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073940    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073943    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073946    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073948    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:03.076580 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073951    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073954    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073956    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073959    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073962    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073964    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073967    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073969    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073972    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073974    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073977    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073980    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073982    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073985    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073987    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073989    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073992    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073994    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.073997    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074001    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:03.077034 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074003    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074006    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074008    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074011    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074013    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074017    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074019    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074022    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074024    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074028    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074030    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074033    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074035    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074038    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074550    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074556    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074560    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074563    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074565    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:03.077541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074568    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074570    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074573    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074576    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074579    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074581    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074584    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074586    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074589    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074592    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074594    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074597    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074601    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074604    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074606    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074609    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074611    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074614    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074618    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:03.077995 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074621    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074623    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074626    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074629    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074631    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074634    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074636    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074639    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074641    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074646    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074650    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074653    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074656    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074659    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074661    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074664    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074666    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074669    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074672    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:03.078467 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074674    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074676    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074679    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074681    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074684    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074686    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074689    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074692    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074694    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074697    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074699    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074702    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074708    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074711    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074713    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074716    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074719    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074721    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074724    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074726    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:03.078906 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074728    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074731    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074733    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074736    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074738    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074741    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074743    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074745    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074748    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074750    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074753    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074756    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074758    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074761    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074763    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074766    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074768    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074771    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074774    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:03.079388 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074776    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074779    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074781    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074783    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074786    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074790    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074793    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074795    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074798    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074801    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074803    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074806    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074808    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074811    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074815    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074817    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074819    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074822    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074824    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:03.079842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074827    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074829    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074832    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074834    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.074837    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074940    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074951    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074961    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074968    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074975    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074979    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074983    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074988    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074991    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074994    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.074997    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075001    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075004    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075008    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075011    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075014    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075017    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075020    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:28:03.080317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075026    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075028    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075032    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075035    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075038    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075043    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075046    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075049    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075052    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075055    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075058    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075061    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075064    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075067    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075072    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075075    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075078    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075082    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075085    2575 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075100    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075105    2575 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075109    2575 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075112    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075115    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:28:03.080861 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075118    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075122    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075125    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075130    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075133    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075136    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075139    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075142    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075145    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075148    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075151    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075153    2575 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075158    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075161    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075164    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075167    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075170    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075173    2575 flags.go:64] FLAG: --help="false"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075176    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075179    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075182    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075185    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075188    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:28:03.081586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075191    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075194    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075197    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075200    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075203    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075206    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075214    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075217    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075220    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075223    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075226    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075229    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075233    2575 flags.go:64] FLAG: --lock-file=""
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075237    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075239    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075242    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075249    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075252    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075254    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075257    2575 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075260    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075264    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075266    2575 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075269    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075273    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:28:03.082163 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075276    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075280    2575 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075283    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075286    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075290    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075292    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075296    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075299    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075302    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075309    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075313    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075316    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075319    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075324    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075327    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075330    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075333    2575 flags.go:64] FLAG: --port="10250"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075336    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075339    2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-05a97ae4f12b6e175"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075343    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075346    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075349    2575 flags.go:64] FLAG: --register-node="true"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075352    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075356    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075359    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075362    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:28:03.082796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075364    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075368    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075371    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075374    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075377    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075380    2575 flags.go:64] FLAG: --runonce="false"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075383    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075386    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075389    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075392    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075395    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075398    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075401    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075404    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075407    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075410    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075412    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075416    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075419    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075423    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075426    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075432    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075435    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075437    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075442    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:28:03.083419 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075446    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075449    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075452    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075455    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075458    2575 flags.go:64] FLAG: --v="2"
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075462    2575 flags.go:64] FLAG: --version="false"
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075466    2575 flags.go:64] FLAG: --vmodule=""
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075471    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075474    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075616    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075620    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075623    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075626    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075629    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075631    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075634    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075636    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075639    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075642    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075645    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075647    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075650    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:03.084010 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075653    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075655    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075658    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075661    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075665    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075668    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075670    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075673    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075675    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075678    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075681    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075684    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075686    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075690    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075694    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075699    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075702    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075706    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:03.084544 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075709    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075712    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075714    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075716    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075719    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075721    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075724    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075726    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075729    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075731    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075733    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075736    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075738    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075741    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075743    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075746    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075748    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075750    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075754    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075757    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:03.085366 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075760    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075762    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075765    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075767    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075770    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075773    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075775    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075778    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075780    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075783    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075785    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075788    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075790    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075793    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075795    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075798    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075800    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075803    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075805    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075808    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:03.086038 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075810    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075812    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075815    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075817    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075820    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075822    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075825    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075827    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075830    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075832    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075836    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075839    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075842    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075844    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075846    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075849    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075852    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075854    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075857    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:03.086558 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075859    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075861    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075864    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075866    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075869    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075871    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075874    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075876    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075879    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075881    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075884    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075886    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075889    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075891    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075894    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075896    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:03.087008 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.075923    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075960    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075965    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075968    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075971    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075974    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075976    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075980    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075982    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075985    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075988    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075991    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075993    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075995    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.075998    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076000    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076003    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076005    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076008    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076010    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:03.087398 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076013    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076015    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076018    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076020    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076023    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076025    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076028    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076030    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076032    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076035    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076037    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076040    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076043    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076046    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076048    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076051    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076055    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076057    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076060    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076064    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:03.087868 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076066    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076069    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076074    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076078    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076083    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076086    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076104    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076106    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076110    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076112    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076114    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076117    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076119    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076122    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076125    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076127    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076130    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076132    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076135    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:03.088385 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076137    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076140    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076143    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076145    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076148    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076150    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076153    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076155    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076158    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076161    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076164    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076166    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076170    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076173    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076175    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076178    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076180    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076183    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076185    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:03.088842 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076188    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076190    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076193    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076195    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076198    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076201    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076203    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076206    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076208    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076210    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076213    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076215    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076218    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076220    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076222    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076225    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076227    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076230    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076232    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076234    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:03.089309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076237    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076239    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076243    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076245    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076248    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076254    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076256    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076259    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.076261    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.083815    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.083831    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.084517    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.084523    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084769    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084782    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084788    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084793    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084805    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084809    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:03.089785 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084814    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084818    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084822    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084826    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084831    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084843    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084849    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084854    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084858    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084862    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084871    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084875    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084879    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084883    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084887    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084892    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084896    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084901    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084905    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084909    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:03.090326 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084913    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084917    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084921    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084931    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084935    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084939    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084943    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084947    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084951    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084955    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084960    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084964    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084970    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084974    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084979    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084983    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084993    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.084997    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085001    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085006    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:03.090784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085011    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085015    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085020    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085024    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085028    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085033    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085037    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085041    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085050    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085054    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085059    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085063    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085067    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085074    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085082    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085088    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085111    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085115    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085119    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:03.091298 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085123    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085129    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085138    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085145    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085150    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085155    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085160    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085164    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085168    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085172    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085176    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085180    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085184    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085189    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085198    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085202    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085206    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085210    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085214    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085218    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:03.091749 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085222    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085227    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085231    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085236    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085240    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085244    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085248    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085258    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085263    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085267    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085271    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085276    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085281    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085285    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085289    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085294    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085298    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085303    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085307    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085316    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:03.092259 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085320    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.085377    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085964    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085976    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085980    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085983    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085987    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085990    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085993    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085995    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.085998    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086001    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086004    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086007    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086010    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086012    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:03.092721 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086015    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086017    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086020    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086023    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086025    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086028    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086030    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086032    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086035    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086038    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086041    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086043    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086046    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086048    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086051    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086054    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086058    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086062    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086066    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:03.093113 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086070    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086073    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086076    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086078    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086081    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086083    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086086    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086105    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086108    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086112    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086115    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086118    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086120    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086125    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086127    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086130    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086132    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086135    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086137    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086140    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:03.093561 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086143    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086145    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086148    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086150    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086153    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086156    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086158    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086161    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086163    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086165    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086168    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086171    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086174    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086177    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086180    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086182    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086186    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086188    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086191    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:03.094037 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086193    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086198    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086202    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086206    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086208    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086211    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086213    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086216    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086218    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086221    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086223    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086226    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086228    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086231    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086233    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086236    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086238    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086241    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086244    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086246    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:03.094579 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086249    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086251    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086253    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086256    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086259    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086262    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086265    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086268    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086270    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086272    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086275    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086277    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086280    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086282    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.086316    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:03.095053 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086377    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086383    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086387    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086390    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086393    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086395    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086398    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086400    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086403    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086405    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086408    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086411    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086413    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086415    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086418    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086420    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086423    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086425    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086428    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086430    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:03.095429 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086432    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086435    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086440    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086443    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086446    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086448    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086451    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086453    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086456    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086459    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086463    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086467    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086470    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086473    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086476    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086479    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086481    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086484    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086487    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:03.095893 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086489    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086492    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086494    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086497    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086499    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086502    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086505    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086507    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086510    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086512    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086515    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086517    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086520    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086522    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086525    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086530    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086532    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086535    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086537    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086540    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:03.096376 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086542    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086545    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086547    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086549    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086552    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086555    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086557    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086560    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086563    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086566    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086570    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086572    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086575    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086577    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086581    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086585    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086588    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086590    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086593    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086596    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:03.096843 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086598    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086601    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086604    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086606    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086609    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086611    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086614    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086619    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086622    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086625    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086628    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086630    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086633    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086635    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086637    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086640    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086642    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086645    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086647    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086650    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:03.097391 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086652    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086655    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086658    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086661    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086663    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086666    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:03.086668    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.086728    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.090076    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.091228    2575 server.go:1014] "Starting client certificate rotation"
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.091329    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.092643    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.094444    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:28:03.097938 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.097024    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:28:03.108776 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.108758    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:28:03.111680 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.111664    2575 log.go:25] "Validated CRI v1 image API"
Nov 03 20:28:03.112626 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.112598    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:28:03.119198 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.119176    2575 fs.go:135] Filesystem UUIDs: map[32f7ec49-494c-4613-87f4-9633b0e626a8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d381774a-e4ab-4d38-b2d2-cf7422b6ee02:/dev/nvme0n1p3]
Nov 03 20:28:03.119259 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.119197    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 20:28:03.125259 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.125138    2575 manager.go:217] Machine: {Timestamp:2026-11-03 20:28:03.123434187 +0000 UTC m=+0.393391920 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3010377 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282563511182b3d5c8715dd68c75ef SystemUUID:ec282563-5111-82b3-d5c8-715dd68c75ef BootID:844b484a-c414-41a1-b6f3-9bf925f732b7 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d3:b4:56:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d3:b4:56:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:33:17:3a:62:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:28:03.125259 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.125258    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:28:03.125404 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.125391    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:28:03.127945 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.127915    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:28:03.128084 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.127947    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-49.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:28:03.128139 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.128086    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:28:03.128139 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.128106    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:28:03.128188 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.128141    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:28:03.128188 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.128158    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:28:03.129222 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.129210    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:28:03.132656 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.132643    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:28:03.132781 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.132772    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:28:03.133596 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.133586    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:28:03.133627 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.133601    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:28:03.133627 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.133612    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:28:03.133627 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.133622    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:28:03.133731 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.133630    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:28:03.133990 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.133978    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:28:03.137735 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.137717    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:28:03.143772 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.143757    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:28:03.144975 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.144963    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:28:03.145014 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.144994    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:28:03.145178 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145168    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:28:03.145224 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145184    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:28:03.145224 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145191    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:28:03.145224 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145197    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:28:03.145224 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145203    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:28:03.145224 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145208    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:28:03.145841 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145832    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:28:03.145868 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145842    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:28:03.145868 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145849    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:28:03.145868 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145855    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:28:03.145868 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145863    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:28:03.146055 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.145873    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:28:03.146973 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.146963    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:28:03.146973 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.146973    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:28:03.150931 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.150915    2575 server.go:1263] "Started kubelet"
Nov 03 20:28:03.151061 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.151020    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:28:03.151454 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.151010    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:28:03.151526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.151483    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:28:03.151773 ip-10-0-128-49 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:28:03.153534 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.153514    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:28:03.154871 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.154853    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:28:03.157842 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.157819    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 20:28:03.160149 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.160119    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:28:03.160985 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.160966    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:28:03.161160 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.161110    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-49.us-west-2.compute.internal\" not found"
Nov 03 20:28:03.161850 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.161832    2575 factory.go:55] Registering systemd factory
Nov 03 20:28:03.161850 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.161851    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:28:03.161850 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.161852    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:28:03.162043 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.161854    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:28:03.162043 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.161875    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:28:03.162043 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.161952    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:28:03.162043 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.161960    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:28:03.162200 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.162073    2575 factory.go:153] Registering CRI-O factory
Nov 03 20:28:03.162200 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.162087    2575 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:28:03.162200 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.162171    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:28:03.162200 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.162191    2575 factory.go:103] Registering Raw factory
Nov 03 20:28:03.162397 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.162205    2575 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:28:03.163310 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.163294    2575 manager.go:319] Starting recovery of all containers
Nov 03 20:28:03.176826 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.176806    2575 manager.go:324] Recovery completed
Nov 03 20:28:03.181586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.181571    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:03.184171 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.184083    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:03.184263 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.184185    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:03.184263 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.184196    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:03.184728 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.184711    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:28:03.184728 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.184724    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:28:03.184846 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.184742    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:28:03.184846 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.184745    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:03.185517 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.185502    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:03.185582 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.185534    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:03.185582 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.185547    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:03.186216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.186203    2575 policy_none.go:50] "Start"
Nov 03 20:28:03.186254 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.186221    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:28:03.186254 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.186240    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:28:03.187502 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.187492    2575 policy_none.go:44] "Start"
Nov 03 20:28:03.219857 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.219668    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:28:03.224154 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.224140    2575 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:28:03.224253 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.224170    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:28:03.224253 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.224179    2575 server.go:87] "Starting device plugin registration server"
Nov 03 20:28:03.224490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.224477    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:28:03.224559 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.224492    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:28:03.224598 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.224585    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:28:03.224697 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.224682    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:28:03.224697 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.224696    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:28:03.225321 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.225299    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:28:03.225412 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.225358    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-49.us-west-2.compute.internal\" not found"
Nov 03 20:28:03.288403 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.288327    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-49.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 20:28:03.288403 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.288323    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-49.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 20:28:03.289411 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.288298    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-49.us-west-2.compute.internal.18e4a2eb5070ce01  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-49.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-49.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:03.150884353 +0000 UTC m=+0.420842086,LastTimestamp:2026-11-03 20:28:03.150884353 +0000 UTC m=+0.420842086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-49.us-west-2.compute.internal,}"
Nov 03 20:28:03.325003 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.324952    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:03.326357 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.326035    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:03.326357 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.326080    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:03.326357 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.326122    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:03.326357 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.326163    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.344315 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.344286    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:28:03.345746 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.345728    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:28:03.345824 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.345754    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:28:03.345824 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.345774    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:28:03.345824 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.345808    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 20:28:03.348563 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.348485    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-49.us-west-2.compute.internal.18e4a2eb526cb2d9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-49.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-49.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-49.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:03.184169689 +0000 UTC m=+0.454127423,LastTimestamp:2026-11-03 20:28:03.184169689 +0000 UTC m=+0.454127423,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-49.us-west-2.compute.internal,}"
Nov 03 20:28:03.352743 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.352724    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:28:03.352854 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.352789    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c5nnf"
Nov 03 20:28:03.446287 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.446240    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"]
Nov 03 20:28:03.446437 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.446317    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:03.447361 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.447338    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:03.447470 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.447374    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:03.447470 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.447385    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:03.448968 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.448953    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:03.449185 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.449142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.449185 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.449178    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:03.449747 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.449731    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:03.449831 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.449755    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:03.449831 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.449758    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:03.449831 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.449770    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:03.449831 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.449777    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:03.449831 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.449789    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:03.451195 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.451181    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.451271 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.451204    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:03.451930 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.451912    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:03.451994 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.451941    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:03.451994 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.451963    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:03.453986 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.453965    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:03.455154 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.455140    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:03.455988 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.455973    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:03.458652 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.458637    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.458737 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.458662    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-49.us-west-2.compute.internal\": node \"ip-10-0-128-49.us-west-2.compute.internal\" not found"
Nov 03 20:28:03.464446 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.464426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.464515 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.464454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.464515 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.464471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0fa7d4d6c53e4753bcd43d896ecfe1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"5b0fa7d4d6c53e4753bcd43d896ecfe1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.488514 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.488478    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-49.us-west-2.compute.internal\" not found" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.492178 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:03.492158    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-49.us-west-2.compute.internal\" not found" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.560001 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.558704    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:03.562099 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.562076    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.565275 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.565255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.565347 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.565284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.565347 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.565300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0fa7d4d6c53e4753bcd43d896ecfe1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"5b0fa7d4d6c53e4753bcd43d896ecfe1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.565418 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.565349    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0fa7d4d6c53e4753bcd43d896ecfe1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"5b0fa7d4d6c53e4753bcd43d896ecfe1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.565418 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.565358    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.565418 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.565367    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.669072 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.669040    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:28:03.670632 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.670617    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.771422 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.771390    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:28:03.793028 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.793005    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:03.796082 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:03.796056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:28:04.091259 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.091232    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 20:28:04.091756 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.091407    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:28:04.091756 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.091407    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:28:04.091756 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.091434    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:28:04.091756 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.091510    2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a6eb617b3f8214fa495946204f12b38d-1c21b587483bc72e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-49.us-west-2.compute.internal.18e4a2eb526cb2d9\": read tcp 10.0.128.49:45174->18.190.119.116:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-49.us-west-2.compute.internal.18e4a2eb526cb2d9  default   4882 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-49.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-49.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-49.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:03 +0000 UTC,LastTimestamp:2026-11-03 20:28:03.447359091 +0000 UTC m=+0.717316826,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-49.us-west-2.compute.internal,}"
Nov 03 20:28:04.134133 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.134111    2575 apiserver.go:52] "Watching apiserver"
Nov 03 20:28:04.160978 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.160955    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 20:28:04.357410 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.357320    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:03 +0000 UTC" deadline="2028-03-29 08:37:05.664180187 +0000 UTC"
Nov 03 20:28:04.357410 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.357353    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12276h9m1.306829855s"
Nov 03 20:28:04.400428 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.400399    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:28:04.400764 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.400740    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-shbqc","kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal","openshift-dns/node-resolver-np7pp","openshift-network-diagnostics/network-check-target-tjt8g","openshift-image-registry/node-ca-nlnjm","openshift-multus/network-metrics-daemon-g67qg","kube-system/konnectivity-agent-bmhst","openshift-cluster-node-tuning-operator/tuned-c6fhq","openshift-multus/multus-8jhfn","openshift-multus/multus-additional-cni-plugins-x5kbn","kube-system/global-pull-secret-syncer-8w5qd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-qmz42"]
Nov 03 20:28:04.402397 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.402377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:04.402488 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.402444    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:04.403625 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.403608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.405191 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.405167    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.406591 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.406574    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.408064 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.408049    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.409565 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.409543    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:04.409647 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.409627    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:04.411033 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.411015    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:04.412345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.412328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.413871 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.413854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.415543 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.415530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.415598 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.415576    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:04.416996 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.416977    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.418616 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.418599    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.457453 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457427    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:28:04.457639 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457613    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:04.457722 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457441    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:28:04.457722 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457458    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:28:04.457722 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457490    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxhx8\""
Nov 03 20:28:04.457722 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457535    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:04.457722 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457594    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8pktm\""
Nov 03 20:28:04.457722 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457597    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:28:04.457722 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.457416    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:28:04.458234 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.458218    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:28:04.458573 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.458550    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:28:04.458757 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.458739    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:28:04.458757 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.458738    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:28:04.458909 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.458835    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:28:04.459066 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.459052    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:28:04.459066 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.459056    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w48hg\""
Nov 03 20:28:04.459932 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.459913    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:28:04.460013 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.459974    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:28:04.460013 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.459984    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:28:04.460106 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.460013    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6vtd\""
Nov 03 20:28:04.462860 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.462840    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:28:04.471034 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-daemon-config\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.471114 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d8dc680-3f86-4ec8-949a-18cb41c680d1-host\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.471114 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:04.471114 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471081    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-cnibin\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.471213 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.471213 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.471270 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcz4p\" (UniqueName: \"kubernetes.io/projected/abf3688b-7862-4d8c-8e8f-03a791523655-kube-api-access-dcz4p\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.471301 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-netns\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471336 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471367 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471334    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-config\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471367 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471362    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-env-overrides\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471421 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovn-node-metrics-cert\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471421 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-netns\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.471518 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471421    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-socket-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.471518 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-var-lib-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471518 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-etc-kubernetes\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.471518 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471510    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-kubelet-config\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.471629 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471532    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xr94\" (UniqueName: \"kubernetes.io/projected/9d8dc680-3f86-4ec8-949a-18cb41c680d1-kube-api-access-2xr94\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.471629 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmp24\" (UniqueName: \"kubernetes.io/projected/492e27da-7982-4035-945f-066588cfdb79-kube-api-access-mmp24\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.471629 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysconfig\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.471629 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-sys\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.471629 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-tuned\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.471767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-slash\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-systemd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2b2l\" (UniqueName: \"kubernetes.io/projected/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-kube-api-access-m2b2l\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-bin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.471767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d8dc680-3f86-4ec8-949a-18cb41c680d1-serviceca\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.471767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-var-lib-kubelet\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.471767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471751    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-tmp\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.471767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-node-log\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-device-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471823    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-systemd\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471846    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471863    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-log-socket\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471883    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-socket-dir-parent\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471899    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-multus\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xkj\" (UniqueName: \"kubernetes.io/projected/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-kube-api-access-c8xkj\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.471972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-script-lib\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492e27da-7982-4035-945f-066588cfdb79-host-slash\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.471998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-kubernetes\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472014    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-k8s-cni-cncf-io\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-hostroot\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-conf-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-conf\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472107    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8gd\" (UniqueName: \"kubernetes.io/projected/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-kube-api-access-xp8gd\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-host\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-system-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-kubelet\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-konnectivity-ca\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-run\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-systemd-units\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472233    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-netd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472247    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmtt5\" (UniqueName: \"kubernetes.io/projected/4b434899-5102-48c6-a2cf-c37ccd545f8a-kube-api-access-mmtt5\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.472282 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-bin\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cnibin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cni-binary-copy\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472309    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-agent-certs\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9zc\" (UniqueName: \"kubernetes.io/projected/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kube-api-access-sl9zc\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-hosts-file\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-lib-modules\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-etc-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-sys-fs\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpl4\" (UniqueName: \"kubernetes.io/projected/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-kube-api-access-nvpl4\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-system-cni-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472551    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-modprobe-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.472730 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-kubelet\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.473179 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.473179 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-multus-certs\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.473179 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472669    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-dbus\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.473179 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-registration-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.473179 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-tmp-dir\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.473179 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/492e27da-7982-4035-945f-066588cfdb79-iptables-alerter-script\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.473179 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.472749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-ovn\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.502917 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.502894    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6582\""
Nov 03 20:28:04.504609 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.504591    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:28:04.504724 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.504608    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mlgd9\""
Nov 03 20:28:04.507360 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.507333    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:28:04.507486 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.507402    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:28:04.507486 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.507426    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:28:04.507486 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.507437    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:28:04.507608 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.507492    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:28:04.507608 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.507507    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:28:04.507608 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.507495    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:28:04.507608 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.507523    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9qmc\""
Nov 03 20:28:04.508062 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.508047    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d54s8\""
Nov 03 20:28:04.508954 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.508935    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:28:04.509336 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.509315    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:04.509422 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.509389    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jb4cj\""
Nov 03 20:28:04.509481 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.509459    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:28:04.509481 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.509472    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:04.509573 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.509538    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tfxf9"
Nov 03 20:28:04.573723 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-k8s-cni-cncf-io\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.573723 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-hostroot\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-conf-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573794    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-k8s-cni-cncf-io\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573809    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-conf-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573803    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-hostroot\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-conf\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8gd\" (UniqueName: \"kubernetes.io/projected/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-kube-api-access-xp8gd\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-host\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.573901 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-system-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573913    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-conf\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-kubelet\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-konnectivity-ca\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-run\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.573992    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-host\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574012    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-system-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574024    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-kubelet\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-systemd-units\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-netd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574128    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-systemd-units\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574131    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-run\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574149    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtt5\" (UniqueName: \"kubernetes.io/projected/4b434899-5102-48c6-a2cf-c37ccd545f8a-kube-api-access-mmtt5\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-bin\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cnibin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574222    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-netd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574252    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cni-binary-copy\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574270    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-bin\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.574333 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574317    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cnibin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-agent-certs\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.574417    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9zc\" (UniqueName: \"kubernetes.io/projected/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kube-api-access-sl9zc\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-hosts-file\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574473    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.574494    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs podName:e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:05.074463474 +0000 UTC m=+2.344421211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs") pod "network-metrics-daemon-g67qg" (UID: "e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574493    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-hosts-file\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-lib-modules\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-etc-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-sys-fs\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpl4\" (UniqueName: \"kubernetes.io/projected/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-kube-api-access-nvpl4\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574725    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-konnectivity-ca\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574721    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-system-cni-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574787    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-lib-modules\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.575216 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574808    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-sys-fs\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574790    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-etc-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574850    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-system-cni-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.574913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-modprobe-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575005    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575020    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-kubelet\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575077    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575076    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-modprobe-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-multus-certs\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575138    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-kubelet\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575147    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575153    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-dbus\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575217    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-dbus\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.576036 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575240    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-registration-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575286    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-multus-certs\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575292    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-registration-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-tmp-dir\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/492e27da-7982-4035-945f-066588cfdb79-iptables-alerter-script\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-ovn\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575363    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cni-binary-copy\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-daemon-config\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d8dc680-3f86-4ec8-949a-18cb41c680d1-host\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575416    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-ovn\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-cnibin\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575494    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-tmp-dir\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575498    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d8dc680-3f86-4ec8-949a-18cb41c680d1-host\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.576885 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575560    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-cnibin\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcz4p\" (UniqueName: \"kubernetes.io/projected/abf3688b-7862-4d8c-8e8f-03a791523655-kube-api-access-dcz4p\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575689    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575699    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-netns\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-config\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-env-overrides\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575816    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/492e27da-7982-4035-945f-066588cfdb79-iptables-alerter-script\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovn-node-metrics-cert\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-netns\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-socket-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-var-lib-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-etc-kubernetes\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-kubelet-config\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.575980    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xr94\" (UniqueName: \"kubernetes.io/projected/9d8dc680-3f86-4ec8-949a-18cb41c680d1-kube-api-access-2xr94\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmp24\" (UniqueName: \"kubernetes.io/projected/492e27da-7982-4035-945f-066588cfdb79-kube-api-access-mmp24\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysconfig\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.577603 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576048    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-daemon-config\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-sys\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576113    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-var-lib-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576123    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-netns\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576125    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-sys\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-tuned\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576196    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-socket-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576193    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576270    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysconfig\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576298    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-etc-kubernetes\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576329    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-kubelet-config\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576434    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-env-overrides\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576448    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-slash\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576424    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-netns\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576477    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576601    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-slash\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-systemd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2b2l\" (UniqueName: \"kubernetes.io/projected/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-kube-api-access-m2b2l\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578295 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576735    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-systemd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576778    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-config\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-bin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576823    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-bin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d8dc680-3f86-4ec8-949a-18cb41c680d1-serviceca\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-var-lib-kubelet\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-tmp\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-node-log\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576941    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-var-lib-kubelet\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-device-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.576996    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-node-log\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-systemd\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577166    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-device-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577172    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577210    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-systemd\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-log-socket\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.578942 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577239    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-socket-dir-parent\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577262    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-log-socket\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-multus\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xkj\" (UniqueName: \"kubernetes.io/projected/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-kube-api-access-c8xkj\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577300    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-socket-dir-parent\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577306    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-multus\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-script-lib\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492e27da-7982-4035-945f-066588cfdb79-host-slash\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-kubernetes\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.577432    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.577484    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret podName:428ef791-c562-4786-b095-7c3b1113bfc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:05.077466607 +0000 UTC m=+2.347424346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret") pod "global-pull-secret-syncer-8w5qd" (UID: "428ef791-c562-4786-b095-7c3b1113bfc8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577508    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-kubernetes\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577559    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492e27da-7982-4035-945f-066588cfdb79-host-slash\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.577560    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.579526 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.578671    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d8dc680-3f86-4ec8-949a-18cb41c680d1-serviceca\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.580000 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.578682    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-script-lib\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.580000 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.579029    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-tuned\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.580000 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.579278    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovn-node-metrics-cert\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.580000 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.579398    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-agent-certs\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:04.580000 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.579610    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-tmp\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.633724 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.633535    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:04.633724 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.633576    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:04.633724 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.633602    2575 projected.go:196] Error preparing data for projected volume kube-api-access-nmqtz for pod openshift-network-diagnostics/network-check-target-tjt8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:04.634355 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.633878    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8gd\" (UniqueName: \"kubernetes.io/projected/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-kube-api-access-xp8gd\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:04.634355 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:04.633984    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz podName:1113fe81-67a7-4c02-ae59-d9fd15e22bd0 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:05.133664643 +0000 UTC m=+2.403622386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nmqtz" (UniqueName: "kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz") pod "network-check-target-tjt8g" (UID: "1113fe81-67a7-4c02-ae59-d9fd15e22bd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:04.637022 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.635097    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcz4p\" (UniqueName: \"kubernetes.io/projected/abf3688b-7862-4d8c-8e8f-03a791523655-kube-api-access-dcz4p\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.637022 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.635151    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9zc\" (UniqueName: \"kubernetes.io/projected/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kube-api-access-sl9zc\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.638727 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.638183    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpl4\" (UniqueName: \"kubernetes.io/projected/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-kube-api-access-nvpl4\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.638727 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.638423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2b2l\" (UniqueName: \"kubernetes.io/projected/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-kube-api-access-m2b2l\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.638727 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.638686    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xr94\" (UniqueName: \"kubernetes.io/projected/9d8dc680-3f86-4ec8-949a-18cb41c680d1-kube-api-access-2xr94\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.639273 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.639245    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmp24\" (UniqueName: \"kubernetes.io/projected/492e27da-7982-4035-945f-066588cfdb79-kube-api-access-mmp24\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:04.639400 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.639327    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtt5\" (UniqueName: \"kubernetes.io/projected/4b434899-5102-48c6-a2cf-c37ccd545f8a-kube-api-access-mmtt5\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.639991 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.639972    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xkj\" (UniqueName: \"kubernetes.io/projected/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-kube-api-access-c8xkj\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.737877 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.737832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:28:04.765633 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.765596    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:28:04.789774 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.789728    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:04.794183 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.794149    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:28:04.807609 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.807576    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:04.814298 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.814266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:28:04.829429 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.829394    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8jhfn"
Nov 03 20:28:04.838069 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.838045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:28:04.854238 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:04.854211    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:28:05.000621 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.000575    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0fa7d4d6c53e4753bcd43d896ecfe1.slice/crio-33e2c1bfb20b02194555408becd8026b73a838f919064c807c2790f11efa3df0 WatchSource:0}: Error finding container 33e2c1bfb20b02194555408becd8026b73a838f919064c807c2790f11efa3df0: Status 404 returned error can't find the container with id 33e2c1bfb20b02194555408becd8026b73a838f919064c807c2790f11efa3df0
Nov 03 20:28:05.001073 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.001040    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77962c08d121df4200a38924d523dec.slice/crio-9ac08017bcf4ab7a41e54278af39a59868dc4eccf639011e5b8f457ac05b2b39 WatchSource:0}: Error finding container 9ac08017bcf4ab7a41e54278af39a59868dc4eccf639011e5b8f457ac05b2b39: Status 404 returned error can't find the container with id 9ac08017bcf4ab7a41e54278af39a59868dc4eccf639011e5b8f457ac05b2b39
Nov 03 20:28:05.005687 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.005607    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:28:05.081310 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.081265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:05.081310 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.081312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:05.081592 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:05.081426    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:05.081592 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:05.081433    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:05.081592 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:05.081483    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs podName:e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:06.081467818 +0000 UTC m=+3.351425553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs") pod "network-metrics-daemon-g67qg" (UID: "e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:05.081592 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:05.081497    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret podName:428ef791-c562-4786-b095-7c3b1113bfc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:06.081490437 +0000 UTC m=+3.351448157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret") pod "global-pull-secret-syncer-8w5qd" (UID: "428ef791-c562-4786-b095-7c3b1113bfc8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:05.182347 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.182299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:05.182990 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:05.182469    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:05.182990 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:05.182493    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:05.182990 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:05.182503    2575 projected.go:196] Error preparing data for projected volume kube-api-access-nmqtz for pod openshift-network-diagnostics/network-check-target-tjt8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:05.182990 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:05.182562    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz podName:1113fe81-67a7-4c02-ae59-d9fd15e22bd0 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:06.182547741 +0000 UTC m=+3.452505476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmqtz" (UniqueName: "kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz") pod "network-check-target-tjt8g" (UID: "1113fe81-67a7-4c02-ae59-d9fd15e22bd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:05.285312 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.285222    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:05.350198 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.350117    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"5b0fa7d4d6c53e4753bcd43d896ecfe1","Type":"ContainerStarted","Data":"33e2c1bfb20b02194555408becd8026b73a838f919064c807c2790f11efa3df0"}
Nov 03 20:28:05.351302 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.351272    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"e77962c08d121df4200a38924d523dec","Type":"ContainerStarted","Data":"9ac08017bcf4ab7a41e54278af39a59868dc4eccf639011e5b8f457ac05b2b39"}
Nov 03 20:28:05.397825 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.397789    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:05.430579 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.430553    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:05.512244 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.512180    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:04 +0000 UTC" deadline="2028-04-14 05:29:59.208149673 +0000 UTC"
Nov 03 20:28:05.512244 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:05.512235    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12657h1m53.6959174s"
Nov 03 20:28:05.676380 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.676351    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b434899_5102_48c6_a2cf_c37ccd545f8a.slice/crio-b0ed02474399c19efdfa535972900ba91d1cdf41ac62fa022dec340e2b56e69e WatchSource:0}: Error finding container b0ed02474399c19efdfa535972900ba91d1cdf41ac62fa022dec340e2b56e69e: Status 404 returned error can't find the container with id b0ed02474399c19efdfa535972900ba91d1cdf41ac62fa022dec340e2b56e69e
Nov 03 20:28:05.695564 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.695533    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf3688b_7862_4d8c_8e8f_03a791523655.slice/crio-25ff54de6d84ef9481b656197a9c26c6809faf59c769cfb8ccf74a7ea156ed4d WatchSource:0}: Error finding container 25ff54de6d84ef9481b656197a9c26c6809faf59c769cfb8ccf74a7ea156ed4d: Status 404 returned error can't find the container with id 25ff54de6d84ef9481b656197a9c26c6809faf59c769cfb8ccf74a7ea156ed4d
Nov 03 20:28:05.732777 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.732745    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d788c1_23d6_4815_8e9f_8d4aebfe1bd3.slice/crio-cf564e89cef3b9789adbccfadee26383b08448b87ddab3cef870381a6d2676b4 WatchSource:0}: Error finding container cf564e89cef3b9789adbccfadee26383b08448b87ddab3cef870381a6d2676b4: Status 404 returned error can't find the container with id cf564e89cef3b9789adbccfadee26383b08448b87ddab3cef870381a6d2676b4
Nov 03 20:28:05.745576 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.745508    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8dc680_3f86_4ec8_949a_18cb41c680d1.slice/crio-e0174d5a15736fa727b3a1ccb3f54e910ae13fca12c0c1abdf516665051a79b2 WatchSource:0}: Error finding container e0174d5a15736fa727b3a1ccb3f54e910ae13fca12c0c1abdf516665051a79b2: Status 404 returned error can't find the container with id e0174d5a15736fa727b3a1ccb3f54e910ae13fca12c0c1abdf516665051a79b2
Nov 03 20:28:05.772328 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.772295    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88128ae2_7d2c_41a4_a0d1_ceb286a7d80f.slice/crio-42807ab1243baaaf868bbd9e9c8f7d44ed57289e344056e2aa66632feb19e2ef WatchSource:0}: Error finding container 42807ab1243baaaf868bbd9e9c8f7d44ed57289e344056e2aa66632feb19e2ef: Status 404 returned error can't find the container with id 42807ab1243baaaf868bbd9e9c8f7d44ed57289e344056e2aa66632feb19e2ef
Nov 03 20:28:05.791784 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.791757    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ee9fcc_846d_4ca8_835f_58d77e3f1a63.slice/crio-0038063b6dccb41ed059b4906cdaa9066b35b3a7b48e4550e165cbb8d954662e WatchSource:0}: Error finding container 0038063b6dccb41ed059b4906cdaa9066b35b3a7b48e4550e165cbb8d954662e: Status 404 returned error can't find the container with id 0038063b6dccb41ed059b4906cdaa9066b35b3a7b48e4550e165cbb8d954662e
Nov 03 20:28:05.823309 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.823276    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3418cdf_4a4a_426f_bdcc_140c84c48a5a.slice/crio-0fc02f488b7cfdccfb2ecf6ddbdaccd8f061781dfc70e205d77d08955c0ea774 WatchSource:0}: Error finding container 0fc02f488b7cfdccfb2ecf6ddbdaccd8f061781dfc70e205d77d08955c0ea774: Status 404 returned error can't find the container with id 0fc02f488b7cfdccfb2ecf6ddbdaccd8f061781dfc70e205d77d08955c0ea774
Nov 03 20:28:05.839676 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.839648    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cabc49_e7ed_4c2a_8652_114a52f0f7a5.slice/crio-3f367a1db172495b6833df570530ad42ec86a1a15b9e829f793190dc2a612074 WatchSource:0}: Error finding container 3f367a1db172495b6833df570530ad42ec86a1a15b9e829f793190dc2a612074: Status 404 returned error can't find the container with id 3f367a1db172495b6833df570530ad42ec86a1a15b9e829f793190dc2a612074
Nov 03 20:28:05.847209 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:05.847178    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492e27da_7982_4035_945f_066588cfdb79.slice/crio-7f141359905d68ece3a56a2690311611483057b67128884a1304e61f0038bb73 WatchSource:0}: Error finding container 7f141359905d68ece3a56a2690311611483057b67128884a1304e61f0038bb73: Status 404 returned error can't find the container with id 7f141359905d68ece3a56a2690311611483057b67128884a1304e61f0038bb73
Nov 03 20:28:06.091155 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.091069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:06.091155 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.091134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:06.091377 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.091248    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:06.091377 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.091316    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs podName:e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:08.091302015 +0000 UTC m=+5.361259735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs") pod "network-metrics-daemon-g67qg" (UID: "e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:06.091377 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.091248    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:06.091377 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.091351    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret podName:428ef791-c562-4786-b095-7c3b1113bfc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:08.091344497 +0000 UTC m=+5.361302271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret") pod "global-pull-secret-syncer-8w5qd" (UID: "428ef791-c562-4786-b095-7c3b1113bfc8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:06.191577 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.191531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:06.192122 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.191744    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:06.192122 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.191768    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:06.192122 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.191778    2575 projected.go:196] Error preparing data for projected volume kube-api-access-nmqtz for pod openshift-network-diagnostics/network-check-target-tjt8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:06.192122 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.191835    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz podName:1113fe81-67a7-4c02-ae59-d9fd15e22bd0 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:08.191820891 +0000 UTC m=+5.461778612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmqtz" (UniqueName: "kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz") pod "network-check-target-tjt8g" (UID: "1113fe81-67a7-4c02-ae59-d9fd15e22bd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:06.346933 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.346830    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:06.347141 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.346980    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:06.347141 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.347077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:06.347274 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.347172    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:06.347274 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.347240    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:06.347376 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:06.347324    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:06.355106 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.355065    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qmz42" event={"ID":"492e27da-7982-4035-945f-066588cfdb79","Type":"ContainerStarted","Data":"7f141359905d68ece3a56a2690311611483057b67128884a1304e61f0038bb73"}
Nov 03 20:28:06.356483 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.356450    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8jhfn" event={"ID":"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5","Type":"ContainerStarted","Data":"3f367a1db172495b6833df570530ad42ec86a1a15b9e829f793190dc2a612074"}
Nov 03 20:28:06.357917 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.357892    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4" event={"ID":"c3418cdf-4a4a-426f-bdcc-140c84c48a5a","Type":"ContainerStarted","Data":"0fc02f488b7cfdccfb2ecf6ddbdaccd8f061781dfc70e205d77d08955c0ea774"}
Nov 03 20:28:06.359643 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.359617    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"42807ab1243baaaf868bbd9e9c8f7d44ed57289e344056e2aa66632feb19e2ef"}
Nov 03 20:28:06.361097 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.361068    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nlnjm" event={"ID":"9d8dc680-3f86-4ec8-949a-18cb41c680d1","Type":"ContainerStarted","Data":"e0174d5a15736fa727b3a1ccb3f54e910ae13fca12c0c1abdf516665051a79b2"}
Nov 03 20:28:06.362567 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.362513    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" event={"ID":"abf3688b-7862-4d8c-8e8f-03a791523655","Type":"ContainerStarted","Data":"25ff54de6d84ef9481b656197a9c26c6809faf59c769cfb8ccf74a7ea156ed4d"}
Nov 03 20:28:06.364014 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.363986    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-np7pp" event={"ID":"53ee9fcc-846d-4ca8-835f-58d77e3f1a63","Type":"ContainerStarted","Data":"0038063b6dccb41ed059b4906cdaa9066b35b3a7b48e4550e165cbb8d954662e"}
Nov 03 20:28:06.365580 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.365554    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bmhst" event={"ID":"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3","Type":"ContainerStarted","Data":"cf564e89cef3b9789adbccfadee26383b08448b87ddab3cef870381a6d2676b4"}
Nov 03 20:28:06.366970 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.366944    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c6fhq" event={"ID":"4b434899-5102-48c6-a2cf-c37ccd545f8a","Type":"ContainerStarted","Data":"b0ed02474399c19efdfa535972900ba91d1cdf41ac62fa022dec340e2b56e69e"}
Nov 03 20:28:06.513066 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.512991    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:04 +0000 UTC" deadline="2028-04-26 08:05:52.371285786 +0000 UTC"
Nov 03 20:28:06.513066 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:06.513061    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12947h37m45.858229944s"
Nov 03 20:28:08.111410 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:08.111364    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:08.111963 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:08.111488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:08.111963 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.111719    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:08.111963 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.111814    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret podName:428ef791-c562-4786-b095-7c3b1113bfc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:12.111783956 +0000 UTC m=+9.381741695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret") pod "global-pull-secret-syncer-8w5qd" (UID: "428ef791-c562-4786-b095-7c3b1113bfc8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:08.112213 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.111975    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:08.112213 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.112040    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs podName:e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:12.112012854 +0000 UTC m=+9.381970577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs") pod "network-metrics-daemon-g67qg" (UID: "e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:08.213013 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:08.212965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:08.213251 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.213180    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:08.213251 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.213203    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:08.213251 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.213217    2575 projected.go:196] Error preparing data for projected volume kube-api-access-nmqtz for pod openshift-network-diagnostics/network-check-target-tjt8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:08.213450 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.213289    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz podName:1113fe81-67a7-4c02-ae59-d9fd15e22bd0 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:12.213269187 +0000 UTC m=+9.483226909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmqtz" (UniqueName: "kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz") pod "network-check-target-tjt8g" (UID: "1113fe81-67a7-4c02-ae59-d9fd15e22bd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:08.346994 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:08.346968    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:08.347154 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.347108    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:08.347274 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:08.347203    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:08.347342 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.347282    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:08.347342 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:08.347337    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:08.347468 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:08.347420    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:09.399768 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:09.399727    2575 generic.go:356] "Generic (PLEG): container finished" podID="e77962c08d121df4200a38924d523dec" containerID="f3335c11d04ad0d5e7a2c7ce98866064a2922aff489d41c9a1af1978467a5fd1" exitCode=0
Nov 03 20:28:09.400227 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:09.399800    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"e77962c08d121df4200a38924d523dec","Type":"ContainerDied","Data":"f3335c11d04ad0d5e7a2c7ce98866064a2922aff489d41c9a1af1978467a5fd1"}
Nov 03 20:28:10.346851 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.346774    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:10.347009 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:10.346928    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:10.347139 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.347012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:10.347139 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:10.347108    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:10.347248 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.347173    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:10.347351 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:10.347256    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:10.422119 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.421316    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"5b0fa7d4d6c53e4753bcd43d896ecfe1","Type":"ContainerStarted","Data":"ec5afee12ba6d33a9f6005389a77c44b572851e89f25d91bab32266920ef012b"}
Nov 03 20:28:10.427987 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.427485    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"e77962c08d121df4200a38924d523dec","Type":"ContainerStarted","Data":"f5b10b1cdbbf287d467a1ec4bfbbbd8c432f76fa55a76f3160f298b39a75bbb2"}
Nov 03 20:28:10.437404 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.436616    2575 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="27264f31fbe7da242def105a6fc72e992aad72fc5f5d0561c899618de9f645c3" exitCode=0
Nov 03 20:28:10.437404 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.436663    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" event={"ID":"abf3688b-7862-4d8c-8e8f-03a791523655","Type":"ContainerDied","Data":"27264f31fbe7da242def105a6fc72e992aad72fc5f5d0561c899618de9f645c3"}
Nov 03 20:28:10.648653 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.648594    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal" podStartSLOduration=6.648573504 podStartE2EDuration="6.648573504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:10.534902508 +0000 UTC m=+7.804860251" watchObservedRunningTime="2026-11-03 20:28:10.648573504 +0000 UTC m=+7.918531245"
Nov 03 20:28:10.768105 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:10.767998    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal" podStartSLOduration=6.767978627 podStartE2EDuration="6.767978627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:10.649424001 +0000 UTC m=+7.919381744" watchObservedRunningTime="2026-11-03 20:28:10.767978627 +0000 UTC m=+8.037936371"
Nov 03 20:28:11.444114 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:11.443668    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bmhst" event={"ID":"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3","Type":"ContainerStarted","Data":"d69f49c094c92a5c2b942e092e9171fc4ee26e9362b122cd474c59246818034e"}
Nov 03 20:28:12.158362 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.158246    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:12.158362 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.158339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:12.158586 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.158485    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:12.158586 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.158546    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret podName:428ef791-c562-4786-b095-7c3b1113bfc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:20.158527713 +0000 UTC m=+17.428485449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret") pod "global-pull-secret-syncer-8w5qd" (UID: "428ef791-c562-4786-b095-7c3b1113bfc8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:12.158729 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.158601    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:12.158729 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.158625    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs podName:e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:20.158617088 +0000 UTC m=+17.428574821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs") pod "network-metrics-daemon-g67qg" (UID: "e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:12.259137 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.259080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:12.259372 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.259347    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:12.259372 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.259372    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:12.259494 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.259386    2575 projected.go:196] Error preparing data for projected volume kube-api-access-nmqtz for pod openshift-network-diagnostics/network-check-target-tjt8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:12.259494 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.259447    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz podName:1113fe81-67a7-4c02-ae59-d9fd15e22bd0 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:20.2594285 +0000 UTC m=+17.529386243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmqtz" (UniqueName: "kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz") pod "network-check-target-tjt8g" (UID: "1113fe81-67a7-4c02-ae59-d9fd15e22bd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:12.346959 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.346500    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:12.346959 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.346633    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:12.346959 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.346703    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:12.346959 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.346779    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:12.346959 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.346843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:12.346959 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:12.346922    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:12.448344 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.448311    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nlnjm" event={"ID":"9d8dc680-3f86-4ec8-949a-18cb41c680d1","Type":"ContainerStarted","Data":"5fbd5e3a643330f72f84eb9dfe466eeb453832e0f06c0fe785d9c81d22ab0ec1"}
Nov 03 20:28:12.560546 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.560482    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nlnjm" podStartSLOduration=3.534918356 podStartE2EDuration="9.560463384s" totalImagesPullingTime="6.025545028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:12.560017422 +0000 UTC m=+9.829975163" watchObservedRunningTime="2026-11-03 20:28:12.560463384 +0000 UTC m=+9.830421127"
Nov 03 20:28:12.560749 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:12.560647    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bmhst" podStartSLOduration=3.833734198 podStartE2EDuration="8.560639158s" totalImagesPullingTime="4.72690496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:11.557661076 +0000 UTC m=+8.827618817" watchObservedRunningTime="2026-11-03 20:28:12.560639158 +0000 UTC m=+9.830596900"
Nov 03 20:28:13.041411 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:13.041379    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:13.041918 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:13.041896    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:13.455004 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:13.454922    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4" event={"ID":"c3418cdf-4a4a-426f-bdcc-140c84c48a5a","Type":"ContainerStarted","Data":"1c4ac32ba52f39b74a4d8819a17ac93ee04077f75189090aba3d4470943904fb"}
Nov 03 20:28:13.455993 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:13.455640    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:13.456668 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:13.456457    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:28:14.346044 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:14.346010    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:14.346467 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:14.346301    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:14.346467 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:14.346387    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:14.346655 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:14.346479    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:14.346655 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:14.346529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:14.346655 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:14.346600    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:14.461325 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:14.461260    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8jhfn" event={"ID":"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5","Type":"ContainerStarted","Data":"510776a4e340095b9df9dce189cb953cdf604a51568126e1094a78a50b645620"}
Nov 03 20:28:14.802552 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:14.802524    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:28:15.234335 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:15.234184    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:28:14.802542412Z","UUID":"5570620b-0182-468e-be05-749776c93278","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:28:15.236069 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:15.236039    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:28:15.236069 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:15.236073    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:28:15.466135 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:15.466028    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4" event={"ID":"c3418cdf-4a4a-426f-bdcc-140c84c48a5a","Type":"ContainerStarted","Data":"9cf97d6b4136a123276a398f2571715a56ec8e008e0055ae21b4373dc1450c54"}
Nov 03 20:28:15.470827 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:15.470786    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-np7pp" event={"ID":"53ee9fcc-846d-4ca8-835f-58d77e3f1a63","Type":"ContainerStarted","Data":"fe6854b8552e61b1e240146e1cee18af634da4d7bb8b6b92871ab48b87dde648"}
Nov 03 20:28:15.584287 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:15.584224    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-np7pp" podStartSLOduration=2.729103433 podStartE2EDuration="11.584203481s" totalImagesPullingTime="8.855100048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:15.583745101 +0000 UTC m=+12.853702910" watchObservedRunningTime="2026-11-03 20:28:15.584203481 +0000 UTC m=+12.854161224"
Nov 03 20:28:15.584546 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:15.584516    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8jhfn" podStartSLOduration=3.084906319 podStartE2EDuration="11.584507667s" totalImagesPullingTime="8.499601348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:14.578644777 +0000 UTC m=+11.848602520" watchObservedRunningTime="2026-11-03 20:28:15.584507667 +0000 UTC m=+12.854465596"
Nov 03 20:28:16.346691 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:16.346600    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:16.346867 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:16.346726    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:16.347162 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:16.347142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:16.347282 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:16.347241    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:16.347347 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:16.347295    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:16.347402 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:16.347376    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:16.487236 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:16.485979    2575 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="2edb00fc77c43e22d753613239afd8e069d7986b48fab760212b772cb255e9b2" exitCode=0
Nov 03 20:28:16.487236 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:16.486937    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" event={"ID":"abf3688b-7862-4d8c-8e8f-03a791523655","Type":"ContainerDied","Data":"2edb00fc77c43e22d753613239afd8e069d7986b48fab760212b772cb255e9b2"}
Nov 03 20:28:17.489853 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:17.489585    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c6fhq" event={"ID":"4b434899-5102-48c6-a2cf-c37ccd545f8a","Type":"ContainerStarted","Data":"27d99d4aed8d881d92739a9313683b78a76dc2c5c2a3f2557ef9c1b1ae18c272"}
Nov 03 20:28:17.491298 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:17.491266    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qmz42" event={"ID":"492e27da-7982-4035-945f-066588cfdb79","Type":"ContainerStarted","Data":"c85cbb385a24adfe8159f875f8fdb5e4c9855c630e0ce6988ad2d38cb25093a3"}
Nov 03 20:28:17.493743 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:17.493716    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4" event={"ID":"c3418cdf-4a4a-426f-bdcc-140c84c48a5a","Type":"ContainerStarted","Data":"2ff7530f2b559d1b21d2387f9580ea923bcf684d789f30d9d8d6410f85f95298"}
Nov 03 20:28:17.721072 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:17.721019    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4" podStartSLOduration=3.525785844 podStartE2EDuration="13.721002799s" totalImagesPullingTime="10.195216955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:17.720899351 +0000 UTC m=+14.990857092" watchObservedRunningTime="2026-11-03 20:28:17.721002799 +0000 UTC m=+14.990960541"
Nov 03 20:28:17.721563 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:17.721535    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c6fhq" podStartSLOduration=3.562489142 podStartE2EDuration="14.721528263s" totalImagesPullingTime="11.159039121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:17.603822426 +0000 UTC m=+14.873780169" watchObservedRunningTime="2026-11-03 20:28:17.721528263 +0000 UTC m=+14.991486004"
Nov 03 20:28:17.837024 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:17.836978    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qmz42" podStartSLOduration=4.538861068 podStartE2EDuration="13.836958515s" totalImagesPullingTime="9.298097447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:17.836767145 +0000 UTC m=+15.106724887" watchObservedRunningTime="2026-11-03 20:28:17.836958515 +0000 UTC m=+15.106916257"
Nov 03 20:28:18.347010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:18.346957    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:18.347010 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:18.346999    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:18.347257 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:18.346999    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:18.347257 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:18.347122    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:18.347257 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:18.347184    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:18.347393 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:18.347266    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:18.498705 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:18.498668    2575 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="96a7b84d3ad84fa8d1ff0486dae1e00f93fde30fa65fd6b44d26ba8cfdf34a1e" exitCode=0
Nov 03 20:28:18.499383 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:18.498769    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" event={"ID":"abf3688b-7862-4d8c-8e8f-03a791523655","Type":"ContainerDied","Data":"96a7b84d3ad84fa8d1ff0486dae1e00f93fde30fa65fd6b44d26ba8cfdf34a1e"}
Nov 03 20:28:20.224307 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:20.224258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:20.225040 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:20.224363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:20.225040 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.224423    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:20.225040 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.224453    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:20.225040 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.224495    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret podName:428ef791-c562-4786-b095-7c3b1113bfc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.224481515 +0000 UTC m=+33.494439247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret") pod "global-pull-secret-syncer-8w5qd" (UID: "428ef791-c562-4786-b095-7c3b1113bfc8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:20.225040 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.224510    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs podName:e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.224504137 +0000 UTC m=+33.494461855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs") pod "network-metrics-daemon-g67qg" (UID: "e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:20.324990 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:20.324952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:20.325162 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.325118    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:20.325162 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.325137    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:20.325162 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.325149    2575 projected.go:196] Error preparing data for projected volume kube-api-access-nmqtz for pod openshift-network-diagnostics/network-check-target-tjt8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:20.325304 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.325225    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz podName:1113fe81-67a7-4c02-ae59-d9fd15e22bd0 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.325199798 +0000 UTC m=+33.595157538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmqtz" (UniqueName: "kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz") pod "network-check-target-tjt8g" (UID: "1113fe81-67a7-4c02-ae59-d9fd15e22bd0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:20.346381 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:20.346350    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:20.346381 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:20.346376    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:20.346607 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:20.346389    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:20.346607 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.346459    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:20.346607 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.346521    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:20.346717 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:20.346605    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:20.506355 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:20.506284    2575 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="3a326b7833effadf3e5e910972ce170538e89f8266be06462cf8fd600f6227d2" exitCode=0
Nov 03 20:28:20.506355 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:20.506330    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" event={"ID":"abf3688b-7862-4d8c-8e8f-03a791523655","Type":"ContainerDied","Data":"3a326b7833effadf3e5e910972ce170538e89f8266be06462cf8fd600f6227d2"}
Nov 03 20:28:21.510959 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:21.510735    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:28:21.511594 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:21.511555    2575 generic.go:356] "Generic (PLEG): container finished" podID="88128ae2-7d2c-41a4-a0d1-ceb286a7d80f" containerID="c752cfa451488c7cb10f22f24b9028101ed1dfb3203e0a9402ec6f8fb693cf07" exitCode=1
Nov 03 20:28:21.511711 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:21.511630    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerDied","Data":"c752cfa451488c7cb10f22f24b9028101ed1dfb3203e0a9402ec6f8fb693cf07"}
Nov 03 20:28:21.511775 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:21.511734    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"07a8684be0642b07f98f25f1adc0a919ccbdba6f5e0f4884451c251b6fc469b9"}
Nov 03 20:28:22.346212 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:22.346177    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:22.346212 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:22.346187    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:22.346460 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:22.346334    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:22.346460 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:22.346332    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:22.346460 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:22.346452    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:22.346587 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:22.346531    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:22.518182 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:22.517937    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:28:22.518641 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:22.518616    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"58745b0d9e246c21624bbea767361b1b81a5e193b40b18f2c18bac46d23a54b1"}
Nov 03 20:28:22.518721 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:22.518706    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"ddb4030242ff7fcf59ada5e14ea59093a25bda2f80892ee2093cbc8b80dc866c"}
Nov 03 20:28:22.518787 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:22.518745    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"16634f14a49439f653ef1884d48d6595bdc0dce3342fd325ff59661a3477c074"}
Nov 03 20:28:22.518843 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:22.518826    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"e74ce9b1c4f844e11eb873897d0a4c92a07b9560c79fb2e0fb0c55bb9490c51e"}
Nov 03 20:28:23.523395 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:23.523356    2575 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="78e53326524745322fe794262d8f9917d985529885ee07d37b13329617185cec" exitCode=0
Nov 03 20:28:23.523395 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:23.523394    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" event={"ID":"abf3688b-7862-4d8c-8e8f-03a791523655","Type":"ContainerDied","Data":"78e53326524745322fe794262d8f9917d985529885ee07d37b13329617185cec"}
Nov 03 20:28:24.346962 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:24.346927    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:24.347166 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:24.347000    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:24.347166 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:24.347027    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:24.347166 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:24.347139    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:24.347332 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:24.347177    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:24.347332 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:24.347243    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:24.528349 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:24.528316    2575 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="a7c8e7db7ac1b281626c79063947765b41935ad15887e7b9b2b608eb58b9d641" exitCode=0
Nov 03 20:28:24.528792 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:24.528400    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" event={"ID":"abf3688b-7862-4d8c-8e8f-03a791523655","Type":"ContainerDied","Data":"a7c8e7db7ac1b281626c79063947765b41935ad15887e7b9b2b608eb58b9d641"}
Nov 03 20:28:24.531609 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:24.531594    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:28:24.531931 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:24.531910    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"9fedbb7b387ad62fc71d1b3eb5a5cc1db07b97270f392441c9e7f9da9f59275b"}
Nov 03 20:28:25.538139 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:25.538110    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" event={"ID":"abf3688b-7862-4d8c-8e8f-03a791523655","Type":"ContainerStarted","Data":"57dbc9a64e22b3cc7f815beb1f7714d168059f303633580921cce516991b2f12"}
Nov 03 20:28:25.755619 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:25.755564    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x5kbn" podStartSLOduration=9.3399525 podStartE2EDuration="22.75555062s" totalImagesPullingTime="13.41559812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:25.755027784 +0000 UTC m=+23.024985526" watchObservedRunningTime="2026-11-03 20:28:25.75555062 +0000 UTC m=+23.025508361"
Nov 03 20:28:26.348923 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:26.346493    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:26.348923 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:26.346542    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:26.348923 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:26.346672    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:26.348923 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:26.347122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:26.348923 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:26.347257    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:26.348923 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:26.347386    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:26.543865 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:26.543841    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:28:26.544577 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:26.544265    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"cbd16c74bdc5843b8f7b38da2f3deec989a051cef684daa55b60a5c90ee2c809"}
Nov 03 20:28:26.544899 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:26.544881    2575 scope.go:122] "RemoveContainer" containerID="c752cfa451488c7cb10f22f24b9028101ed1dfb3203e0a9402ec6f8fb693cf07"
Nov 03 20:28:27.552779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:27.552556    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:28:27.553306 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:27.553279    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" event={"ID":"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f","Type":"ContainerStarted","Data":"af7a7a5bf96731e715c13c181f27220a59652c4d2e2271e383cb5b83add919be"}
Nov 03 20:28:27.553529 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:27.553511    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:27.553667 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:27.553656    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:27.553793 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:27.553784    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:27.568998 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:27.568974    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:27.569152 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:27.569072    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:27.828663 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:27.828616    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc" podStartSLOduration=9.380297465 podStartE2EDuration="24.82860274s" totalImagesPullingTime="15.448305275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:27.828153809 +0000 UTC m=+25.098111550" watchObservedRunningTime="2026-11-03 20:28:27.82860274 +0000 UTC m=+25.098560481"
Nov 03 20:28:28.346447 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.346415    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:28.346447 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.346436    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:28.346675 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.346436    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:28.346675 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:28.346515    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:28.347026 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:28.346996    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:28.347220 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:28.347132    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:28.627037 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.626954    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8w5qd"]
Nov 03 20:28:28.627466 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.627129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:28.627466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:28.627221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:28.629793 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.629769    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g67qg"]
Nov 03 20:28:28.629944 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.629904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:28.630029 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:28.630010    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:28.630357 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.630336    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tjt8g"]
Nov 03 20:28:28.630434 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:28.630424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:28.630500 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:28.630485    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:30.345991 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:30.345956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:30.346522 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:30.346052    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:30.346522 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:30.346062    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:30.346522 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:30.346108    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:30.346522 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:30.346188    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:30.346522 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:30.346264    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:32.346921 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:32.346890    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:32.347331 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:32.346930    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:32.347331 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:32.346970    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:32.347331 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:32.347016    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:32.347331 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:32.347126    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:32.347331 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:32.347193    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:34.346290 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.346252    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:34.346290 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.346284    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:34.346703 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.346366    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:34.346703 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:34.346372    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjt8g" podUID="1113fe81-67a7-4c02-ae59-d9fd15e22bd0"
Nov 03 20:28:34.346703 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:34.346435    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w5qd" podUID="428ef791-c562-4786-b095-7c3b1113bfc8"
Nov 03 20:28:34.346703 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:34.346518    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g67qg" podUID="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6"
Nov 03 20:28:34.525563 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.525525    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeReady"
Nov 03 20:28:34.525749 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.525654    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:28:34.637507 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.637444    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeReady"
Nov 03 20:28:34.681784 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.681754    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6vt8w"]
Nov 03 20:28:34.706556 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.706526    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hjc54"]
Nov 03 20:28:34.706722 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.706661    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6vt8w"
Nov 03 20:28:34.722750 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.722592    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qpwf2"]
Nov 03 20:28:34.722848 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.722705    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hjc54"
Nov 03 20:28:34.740250 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.740221    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"]
Nov 03 20:28:34.740379 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.740346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qpwf2"
Nov 03 20:28:34.755186 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.755165    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c7454997c-ccrh5"]
Nov 03 20:28:34.755300 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.755283    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.759143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.759124    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 20:28:34.779284 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.779017    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"]
Nov 03 20:28:34.779284 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.779063    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:34.782248 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.782225    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 20:28:34.799339 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.799304    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-g2rqz"]
Nov 03 20:28:34.799472 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.799430    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:34.807790 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.807762    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 20:28:34.814584 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.814562    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"]
Nov 03 20:28:34.814693 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.814671    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:34.828589 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.828569    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"]
Nov 03 20:28:34.828718 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.828695    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:34.828783 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.828762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt5pj\" (UniqueName: \"kubernetes.io/projected/68c418d8-0eb1-44a4-9c0f-59d3d7d724af-kube-api-access-xt5pj\") pod \"kas-connection-checker-685895c7c9-hjc54\" (UID: \"68c418d8-0eb1-44a4-9c0f-59d3d7d724af\") " pod="kube-system/kas-connection-checker-685895c7c9-hjc54"
Nov 03 20:28:34.828836 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.828801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.828909 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.828887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqk6q\" (UniqueName: \"kubernetes.io/projected/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-kube-api-access-mqk6q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.829009 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.828978    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpgb4\" (UniqueName: \"kubernetes.io/projected/1d9f21de-9ec8-48b7-aa66-5945a79d95e4-kube-api-access-rpgb4\") pod \"kas-connection-checker-685895c7c9-6vt8w\" (UID: \"1d9f21de-9ec8-48b7-aa66-5945a79d95e4\") " pod="kube-system/kas-connection-checker-685895c7c9-6vt8w"
Nov 03 20:28:34.829131 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.829079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqjh\" (UniqueName: \"kubernetes.io/projected/a533118b-20c1-478a-b7e7-42dd7b4f200c-kube-api-access-szqjh\") pod \"kas-connection-checker-685895c7c9-qpwf2\" (UID: \"a533118b-20c1-478a-b7e7-42dd7b4f200c\") " pod="kube-system/kas-connection-checker-685895c7c9-qpwf2"
Nov 03 20:28:34.829211 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.829161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.852801 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.852776    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fbd46f478-8pr5x"]
Nov 03 20:28:34.852915 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.852897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:34.859800 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.859776    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 20:28:34.859800 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.859785    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 20:28:34.859800 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.859807    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 20:28:34.860001 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.859808    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 20:28:34.860001 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.859808    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 20:28:34.860001 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.859809    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:34.860001 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.859879    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:34.866858 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.866837    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:28:34.866858 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.866838    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:28:34.867022 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.866903    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 20:28:34.871206 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.870278    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 20:28:34.871592 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.871574    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v455q"]
Nov 03 20:28:34.871767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.871751    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:34.881064 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.881045    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:28:34.881204 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.881120    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:28:34.894134 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.894074    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"]
Nov 03 20:28:34.894219 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.894173    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:34.908327 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.908307    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-567984ddc5-9dl9n"]
Nov 03 20:28:34.908438 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.908423    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"
Nov 03 20:28:34.917417 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917387    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 20:28:34.917417 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917402    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 20:28:34.917555 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917418    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 20:28:34.917555 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917428    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 20:28:34.917555 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917438    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 20:28:34.917555 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917449    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 20:28:34.917555 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917476    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 20:28:34.917555 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917501    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 20:28:34.917738 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.917577    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-b585d\""
Nov 03 20:28:34.926999 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.926981    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 20:28:34.927165 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.927151    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 20:28:34.927221 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.927201    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwh5m\""
Nov 03 20:28:34.927269 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.927253    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 20:28:34.929570 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt5pj\" (UniqueName: \"kubernetes.io/projected/68c418d8-0eb1-44a4-9c0f-59d3d7d724af-kube-api-access-xt5pj\") pod \"kas-connection-checker-685895c7c9-hjc54\" (UID: \"68c418d8-0eb1-44a4-9c0f-59d3d7d724af\") " pod="kube-system/kas-connection-checker-685895c7c9-hjc54"
Nov 03 20:28:34.929618 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929581    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:34.929618 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:34.929672 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-service-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:34.929704 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.929763 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3946b8e3-670c-407b-ba35-01cd37890b2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:34.929816 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqk6q\" (UniqueName: \"kubernetes.io/projected/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-kube-api-access-mqk6q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.929858 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlcpg\" (UniqueName: \"kubernetes.io/projected/151b0e6d-c5a6-469c-8d77-f7507ec6796a-kube-api-access-nlcpg\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:34.929921 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929899    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwc28\" (UniqueName: \"kubernetes.io/projected/e8b325b4-c286-465f-83c9-623e0d441d17-kube-api-access-fwc28\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:34.929960 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpgb4\" (UniqueName: \"kubernetes.io/projected/1d9f21de-9ec8-48b7-aa66-5945a79d95e4-kube-api-access-rpgb4\") pod \"kas-connection-checker-685895c7c9-6vt8w\" (UID: \"1d9f21de-9ec8-48b7-aa66-5945a79d95e4\") " pod="kube-system/kas-connection-checker-685895c7c9-6vt8w"
Nov 03 20:28:34.929997 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.929965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8sqv\" (UniqueName: \"kubernetes.io/projected/ee248b83-7c43-46fe-809c-69b61ca6254a-kube-api-access-s8sqv\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:34.930063 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930032    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3b4704b-2627-4530-8100-838f2d8b7d31-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:34.930158 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:34.930225 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:34.930270 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930227    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.930270 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqjh\" (UniqueName: \"kubernetes.io/projected/a533118b-20c1-478a-b7e7-42dd7b4f200c-kube-api-access-szqjh\") pod \"kas-connection-checker-685895c7c9-qpwf2\" (UID: \"a533118b-20c1-478a-b7e7-42dd7b4f200c\") " pod="kube-system/kas-connection-checker-685895c7c9-qpwf2"
Nov 03 20:28:34.930354 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-tmp\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:34.930354 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930290    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-snapshots\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:34.930354 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930328    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-default-certificate\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:34.930465 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-stats-auth\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:34.930465 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151b0e6d-c5a6-469c-8d77-f7507ec6796a-serving-cert\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:34.930465 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:34.930465 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b325b4-c286-465f-83c9-623e0d441d17-serving-cert\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:34.930586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b325b4-c286-465f-83c9-623e0d441d17-config\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:34.930586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.930586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.930568    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcfn\" (UniqueName: \"kubernetes.io/projected/a3b4704b-2627-4530-8100-838f2d8b7d31-kube-api-access-fdcfn\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:34.932377 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.932362    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"]
Nov 03 20:28:34.932486 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.932474    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:34.934336 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.934316    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.946256 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.946237    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"]
Nov 03 20:28:34.946365 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.946352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:34.959926 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.959909    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 20:28:34.960011 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.959961    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:34.960011 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.959995    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:34.960088 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.960036    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:34.960088 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.960078    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:34.960238 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.960085    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 20:28:34.961622 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.961577    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:34.961622 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.961611    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:34.962552 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.962531    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 20:28:34.962616 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.962597    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 20:28:34.964599 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.964580    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hpchn"]
Nov 03 20:28:34.964713 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.964697    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"
Nov 03 20:28:34.979604 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.979583    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k9c8l"]
Nov 03 20:28:34.979696 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.979679    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:28:34.987765 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.987741    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqk6q\" (UniqueName: \"kubernetes.io/projected/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-kube-api-access-mqk6q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:34.987929 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.987913    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt5pj\" (UniqueName: \"kubernetes.io/projected/68c418d8-0eb1-44a4-9c0f-59d3d7d724af-kube-api-access-xt5pj\") pod \"kas-connection-checker-685895c7c9-hjc54\" (UID: \"68c418d8-0eb1-44a4-9c0f-59d3d7d724af\") " pod="kube-system/kas-connection-checker-685895c7c9-hjc54"
Nov 03 20:28:34.987990 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.987913    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpgb4\" (UniqueName: \"kubernetes.io/projected/1d9f21de-9ec8-48b7-aa66-5945a79d95e4-kube-api-access-rpgb4\") pod \"kas-connection-checker-685895c7c9-6vt8w\" (UID: \"1d9f21de-9ec8-48b7-aa66-5945a79d95e4\") " pod="kube-system/kas-connection-checker-685895c7c9-6vt8w"
Nov 03 20:28:34.987990 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:34.987958    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqjh\" (UniqueName: \"kubernetes.io/projected/a533118b-20c1-478a-b7e7-42dd7b4f200c-kube-api-access-szqjh\") pod \"kas-connection-checker-685895c7c9-qpwf2\" (UID: \"a533118b-20c1-478a-b7e7-42dd7b4f200c\") " pod="kube-system/kas-connection-checker-685895c7c9-qpwf2"
Nov 03 20:28:35.005586 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005561    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hjc54"]
Nov 03 20:28:35.005696 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005601    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6vt8w"]
Nov 03 20:28:35.005696 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005624    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qpwf2"]
Nov 03 20:28:35.005696 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005643    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"]
Nov 03 20:28:35.005696 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005664    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"]
Nov 03 20:28:35.005696 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005681    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.005851 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005685    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"]
Nov 03 20:28:35.005851 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005826    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"]
Nov 03 20:28:35.005907 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005856    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-g2rqz"]
Nov 03 20:28:35.005907 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005885    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"]
Nov 03 20:28:35.005907 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005906    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hpchn"]
Nov 03 20:28:35.005988 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005924    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9c8l"]
Nov 03 20:28:35.005988 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005944    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-567984ddc5-9dl9n"]
Nov 03 20:28:35.005988 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005962    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"]
Nov 03 20:28:35.005988 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005981    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c7454997c-ccrh5"]
Nov 03 20:28:35.006121 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.005993    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"]
Nov 03 20:28:35.006121 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.006006    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fbd46f478-8pr5x"]
Nov 03 20:28:35.006121 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.006019    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v455q"]
Nov 03 20:28:35.008176 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.008155    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 20:28:35.008176 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.008171    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 20:28:35.029634 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.029606    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6vt8w"
Nov 03 20:28:35.031953 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.031929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvg9\" (UniqueName: \"kubernetes.io/projected/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a-kube-api-access-7wvg9\") pod \"volume-data-source-validator-58c9f5f46d-hswtf\" (UID: \"859d83db-cfcc-459f-8ed2-8ec6a7e88b1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"
Nov 03 20:28:35.031953 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.031936    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:28:35.032076 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032001    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fpmzx\""
Nov 03 20:28:35.032076 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkz4p\" (UniqueName: \"kubernetes.io/projected/68f3b7ee-a558-4c4d-8625-11f72d13b257-kube-api-access-qkz4p\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.032076 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpww\" (UniqueName: \"kubernetes.io/projected/170428c7-02e4-40f5-a88c-1634d8b58b5a-kube-api-access-8dpww\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.032076 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7nw\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-kube-api-access-5f7nw\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.032211 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032125    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.032211 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-bound-sa-token\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.032211 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlcpg\" (UniqueName: \"kubernetes.io/projected/151b0e6d-c5a6-469c-8d77-f7507ec6796a-kube-api-access-nlcpg\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.032294 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwc28\" (UniqueName: \"kubernetes.io/projected/e8b325b4-c286-465f-83c9-623e0d441d17-kube-api-access-fwc28\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:35.032294 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032271    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-trusted-ca\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.032294 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032290    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-image-registry-private-configuration\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.032398 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-image-registry-private-configuration\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.032398 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032337    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-trusted-ca\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.032398 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.032477 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.032477 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.032539 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032480    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.032569 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.032549    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 20:28:35.032617 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.032605    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:35.532585753 +0000 UTC m=+32.802543505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : secret "router-metrics-certs-default" not found
Nov 03 20:28:35.032688 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032673    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3946b8e3-670c-407b-ba35-01cd37890b2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:35.032725 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151b0e6d-c5a6-469c-8d77-f7507ec6796a-serving-cert\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.032756 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-tmp\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.032789 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-stats-auth\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.032834 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b325b4-c286-465f-83c9-623e0d441d17-serving-cert\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:35.032834 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032820    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2xz\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-kube-api-access-rj2xz\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.032928 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-config\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.032928 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7c1d3a1-ac62-4cce-950a-640c97725c08-ca-trust-extracted\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.032928 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-trusted-ca\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.032928 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.033143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-certificates\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.033143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.032982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8sqv\" (UniqueName: \"kubernetes.io/projected/ee248b83-7c43-46fe-809c-69b61ca6254a-kube-api-access-s8sqv\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.033143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033003    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/170428c7-02e4-40f5-a88c-1634d8b58b5a-serving-cert\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.033143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-service-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.033143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3b4704b-2627-4530-8100-838f2d8b7d31-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:35.033143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:35.033143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033125    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-tmp\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-installation-pull-secrets\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-installation-pull-secrets\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcfn\" (UniqueName: \"kubernetes.io/projected/a3b4704b-2627-4530-8100-838f2d8b7d31-kube-api-access-fdcfn\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-snapshots\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-ca-trust-extracted\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033378    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-certificates\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-bound-sa-token\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033422    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033425    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3946b8e3-670c-407b-ba35-01cd37890b2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b325b4-c286-465f-83c9-623e0d441d17-config\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:35.033490 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033486    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmt7s\" (UniqueName: \"kubernetes.io/projected/17826854-67bc-4852-97c1-83ebe2c648f7-kube-api-access-wmt7s\") pod \"network-check-source-6f78d59f99-hpm76\" (UID: \"17826854-67bc-4852-97c1-83ebe2c648f7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"
Nov 03 20:28:35.034058 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.033516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-default-certificate\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.034058 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.033727    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:35.034058 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.033778    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls podName:a3b4704b-2627-4530-8100-838f2d8b7d31 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:35.533761695 +0000 UTC m=+32.803719417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4g2kg" (UID: "a3b4704b-2627-4530-8100-838f2d8b7d31") : secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:35.034241 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.034154    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-snapshots\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.034322 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.034257    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 20:28:35.034322 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.034306    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert podName:3946b8e3-670c-407b-ba35-01cd37890b2d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:35.534290383 +0000 UTC m=+32.804248121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q6fg" (UID: "3946b8e3-670c-407b-ba35-01cd37890b2d") : secret "networking-console-plugin-cert" not found
Nov 03 20:28:35.034600 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.034580    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3b4704b-2627-4530-8100-838f2d8b7d31-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:35.034680 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.034590    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:35.534575161 +0000 UTC m=+32.804532916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : configmap references non-existent config key: service-ca.crt
Nov 03 20:28:35.035116 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.035074    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-service-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.035222 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.035182    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b325b4-c286-465f-83c9-623e0d441d17-config\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:35.035608 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.035582    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-stats-auth\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.035689 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.035644    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151b0e6d-c5a6-469c-8d77-f7507ec6796a-serving-cert\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.035726 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.035710    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b325b4-c286-465f-83c9-623e0d441d17-serving-cert\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:35.036218 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.036201    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-default-certificate\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.048981 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.048961    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hjc54"
Nov 03 20:28:35.060274 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.060250    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:35.060384 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.060335    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 20:28:35.060447 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.060385    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:35.060447 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.060418    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:28:35.060447 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.060434    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:28:35.060557 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.060471    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:28:35.060557 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.060482    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:28:35.060557 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.060472    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:28:35.069880 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.069855    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qpwf2"
Nov 03 20:28:35.089321 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.089297    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:28:35.093903 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.092655    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwc28\" (UniqueName: \"kubernetes.io/projected/e8b325b4-c286-465f-83c9-623e0d441d17-kube-api-access-fwc28\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:35.093903 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.093268    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlcpg\" (UniqueName: \"kubernetes.io/projected/151b0e6d-c5a6-469c-8d77-f7507ec6796a-kube-api-access-nlcpg\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.093903 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.093851    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcfn\" (UniqueName: \"kubernetes.io/projected/a3b4704b-2627-4530-8100-838f2d8b7d31-kube-api-access-fdcfn\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:35.097034 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.096465    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8sqv\" (UniqueName: \"kubernetes.io/projected/ee248b83-7c43-46fe-809c-69b61ca6254a-kube-api-access-s8sqv\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.108781 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.108751    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:28:35.108913 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.108784    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pqh8h\""
Nov 03 20:28:35.134560 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-trusted-ca\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.134653 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-image-registry-private-configuration\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.134653 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-image-registry-private-configuration\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.134653 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ssd9\" (UniqueName: \"kubernetes.io/projected/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-kube-api-access-5ssd9\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.134790 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-trusted-ca\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.134790 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.134790 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.134790 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2xz\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-kube-api-access-rj2xz\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-config\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfdf39cf-ae26-400e-b929-80512e786f5a-cert\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7c1d3a1-ac62-4cce-950a-640c97725c08-ca-trust-extracted\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-trusted-ca\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134936    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-certificates\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134954    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpqfx\" (UniqueName: \"kubernetes.io/projected/bfdf39cf-ae26-400e-b929-80512e786f5a-kube-api-access-tpqfx\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:28:35.134976 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.134977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/170428c7-02e4-40f5-a88c-1634d8b58b5a-serving-cert\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-installation-pull-secrets\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-installation-pull-secrets\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135067    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-ca-trust-extracted\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-certificates\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-bound-sa-token\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmt7s\" (UniqueName: \"kubernetes.io/projected/17826854-67bc-4852-97c1-83ebe2c648f7-kube-api-access-wmt7s\") pod \"network-check-source-6f78d59f99-hpm76\" (UID: \"17826854-67bc-4852-97c1-83ebe2c648f7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135239    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-tmp-dir\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvg9\" (UniqueName: \"kubernetes.io/projected/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a-kube-api-access-7wvg9\") pod \"volume-data-source-validator-58c9f5f46d-hswtf\" (UID: \"859d83db-cfcc-459f-8ed2-8ec6a7e88b1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"
Nov 03 20:28:35.135345 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkz4p\" (UniqueName: \"kubernetes.io/projected/68f3b7ee-a558-4c4d-8625-11f72d13b257-kube-api-access-qkz4p\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.135584 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-config-volume\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.135584 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpww\" (UniqueName: \"kubernetes.io/projected/170428c7-02e4-40f5-a88c-1634d8b58b5a-kube-api-access-8dpww\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.135584 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7nw\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-kube-api-access-5f7nw\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.135584 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135452    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.135584 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135477    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-bound-sa-token\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.135584 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135478    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-trusted-ca\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.136086 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.135778    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-ca-trust-extracted\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.136166 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.136113    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-config\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.136349 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.136329    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:35.136404 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.136351    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd46f478-8pr5x: secret "image-registry-tls" not found
Nov 03 20:28:35.136454 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.136405    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls podName:a7c1d3a1-ac62-4cce-950a-640c97725c08 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:35.636386944 +0000 UTC m=+32.906344681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls") pod "image-registry-fbd46f478-8pr5x" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08") : secret "image-registry-tls" not found
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.137253    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.137380    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-trusted-ca\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.137467    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.137524    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls podName:68f3b7ee-a558-4c4d-8625-11f72d13b257 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:35.63750668 +0000 UTC m=+32.907464401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zd5zj" (UID: "68f3b7ee-a558-4c4d-8625-11f72d13b257") : secret "samples-operator-tls" not found
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.137557    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7c1d3a1-ac62-4cce-950a-640c97725c08-ca-trust-extracted\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.137929    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.137946    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567984ddc5-9dl9n: secret "image-registry-tls" not found
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.138344    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls podName:33bbc6dc-efa5-41c9-a6c6-d3a37542ea92 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:35.638323652 +0000 UTC m=+32.908281379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls") pod "image-registry-567984ddc5-9dl9n" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92") : secret "image-registry-tls" not found
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.138339    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-trusted-ca\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.138514    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-image-registry-private-configuration\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.138585    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/170428c7-02e4-40f5-a88c-1634d8b58b5a-serving-cert\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.138779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.138812    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-certificates\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.139276 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.138880    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-certificates\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.140152 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.139280    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-installation-pull-secrets\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.140775 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.140754    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-image-registry-private-configuration\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.141644 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.141628    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-installation-pull-secrets\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.143950 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.143931    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:28:35.185513 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.185449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:28:35.222299 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.221836    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-bound-sa-token\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.222299 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.222039    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpww\" (UniqueName: \"kubernetes.io/projected/170428c7-02e4-40f5-a88c-1634d8b58b5a-kube-api-access-8dpww\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.223256 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.223046    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-bound-sa-token\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.223256 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.223199    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:35.224453 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.224409    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7nw\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-kube-api-access-5f7nw\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.224869 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.224811    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkz4p\" (UniqueName: \"kubernetes.io/projected/68f3b7ee-a558-4c4d-8625-11f72d13b257-kube-api-access-qkz4p\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.226068 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.226014    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvg9\" (UniqueName: \"kubernetes.io/projected/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a-kube-api-access-7wvg9\") pod \"volume-data-source-validator-58c9f5f46d-hswtf\" (UID: \"859d83db-cfcc-459f-8ed2-8ec6a7e88b1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"
Nov 03 20:28:35.226423 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.226376    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2xz\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-kube-api-access-rj2xz\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.226506 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.225966    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmt7s\" (UniqueName: \"kubernetes.io/projected/17826854-67bc-4852-97c1-83ebe2c648f7-kube-api-access-wmt7s\") pod \"network-check-source-6f78d59f99-hpm76\" (UID: \"17826854-67bc-4852-97c1-83ebe2c648f7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"
Nov 03 20:28:35.236972 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.236924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpqfx\" (UniqueName: \"kubernetes.io/projected/bfdf39cf-ae26-400e-b929-80512e786f5a-kube-api-access-tpqfx\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:28:35.237057 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.237030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-tmp-dir\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.237121 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.237062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-config-volume\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.237171 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.237120    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssd9\" (UniqueName: \"kubernetes.io/projected/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-kube-api-access-5ssd9\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.237236 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.237187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.237446 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.237240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfdf39cf-ae26-400e-b929-80512e786f5a-cert\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:28:35.237914 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.237890    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-config-volume\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.238169 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.238145    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-tmp-dir\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.238280 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.238264    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:28:35.238339 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.238312    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls podName:545aca01-0ea8-4df2-b3bc-fa60f6f1c725 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:35.738299311 +0000 UTC m=+33.008257030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls") pod "dns-default-k9c8l" (UID: "545aca01-0ea8-4df2-b3bc-fa60f6f1c725") : secret "dns-default-metrics-tls" not found
Nov 03 20:28:35.243268 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.243216    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfdf39cf-ae26-400e-b929-80512e786f5a-cert\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:28:35.243374 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.243346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"
Nov 03 20:28:35.298533 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.297806    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssd9\" (UniqueName: \"kubernetes.io/projected/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-kube-api-access-5ssd9\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.298636 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.298544    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpqfx\" (UniqueName: \"kubernetes.io/projected/bfdf39cf-ae26-400e-b929-80512e786f5a-kube-api-access-tpqfx\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:28:35.316650 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.316612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:28:35.340633 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.338006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"
Nov 03 20:28:35.532309 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.532277    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6vt8w"]
Nov 03 20:28:35.533485 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.533452    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qpwf2"]
Nov 03 20:28:35.536072 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.536047    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9f21de_9ec8_48b7_aa66_5945a79d95e4.slice/crio-effde35f08fb9873c7d2e4336acffd86d749d9008847ef31adca6c8cc5882f9b WatchSource:0}: Error finding container effde35f08fb9873c7d2e4336acffd86d749d9008847ef31adca6c8cc5882f9b: Status 404 returned error can't find the container with id effde35f08fb9873c7d2e4336acffd86d749d9008847ef31adca6c8cc5882f9b
Nov 03 20:28:35.536881 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.536849    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda533118b_20c1_478a_b7e7_42dd7b4f200c.slice/crio-a138a24953b025e246d185cb92a3f4476bbba2e1df99bcd83ada96f5bf60d382 WatchSource:0}: Error finding container a138a24953b025e246d185cb92a3f4476bbba2e1df99bcd83ada96f5bf60d382: Status 404 returned error can't find the container with id a138a24953b025e246d185cb92a3f4476bbba2e1df99bcd83ada96f5bf60d382
Nov 03 20:28:35.538701 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.538680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.538807 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.538760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:35.538866 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.538806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:35.538866 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.538832    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 20:28:35.538866 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.538843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:35.538994 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.538892    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.538869446 +0000 UTC m=+33.808827184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : secret "router-metrics-certs-default" not found
Nov 03 20:28:35.538994 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.538913    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 20:28:35.538994 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.538935    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:35.538994 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.538956    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.538937983 +0000 UTC m=+33.808895738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : configmap references non-existent config key: service-ca.crt
Nov 03 20:28:35.538994 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.538982    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert podName:3946b8e3-670c-407b-ba35-01cd37890b2d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.538971059 +0000 UTC m=+33.808928784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q6fg" (UID: "3946b8e3-670c-407b-ba35-01cd37890b2d") : secret "networking-console-plugin-cert" not found
Nov 03 20:28:35.539277 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.538998    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls podName:a3b4704b-2627-4530-8100-838f2d8b7d31 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.53898994 +0000 UTC m=+33.808947665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4g2kg" (UID: "a3b4704b-2627-4530-8100-838f2d8b7d31") : secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:35.572484 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.572429    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qpwf2" event={"ID":"a533118b-20c1-478a-b7e7-42dd7b4f200c","Type":"ContainerStarted","Data":"a138a24953b025e246d185cb92a3f4476bbba2e1df99bcd83ada96f5bf60d382"}
Nov 03 20:28:35.574190 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.574160    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6vt8w" event={"ID":"1d9f21de-9ec8-48b7-aa66-5945a79d95e4","Type":"ContainerStarted","Data":"effde35f08fb9873c7d2e4336acffd86d749d9008847ef31adca6c8cc5882f9b"}
Nov 03 20:28:35.582156 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.582131    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"]
Nov 03 20:28:35.586241 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.586204    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9c06cb_46f7_440b_b395_a4c4e9a597cd.slice/crio-04eba560ee1d9cc80f006d14e5af55b65c817d405a813190c4f6f432e63186d4 WatchSource:0}: Error finding container 04eba560ee1d9cc80f006d14e5af55b65c817d405a813190c4f6f432e63186d4: Status 404 returned error can't find the container with id 04eba560ee1d9cc80f006d14e5af55b65c817d405a813190c4f6f432e63186d4
Nov 03 20:28:35.599283 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.599252    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-g2rqz"]
Nov 03 20:28:35.604046 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.604017    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151b0e6d_c5a6_469c_8d77_f7507ec6796a.slice/crio-37c190292dca8dd29400317822932d7169a10879152a603eab7442e65203d8c3 WatchSource:0}: Error finding container 37c190292dca8dd29400317822932d7169a10879152a603eab7442e65203d8c3: Status 404 returned error can't find the container with id 37c190292dca8dd29400317822932d7169a10879152a603eab7442e65203d8c3
Nov 03 20:28:35.632743 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.632717    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"]
Nov 03 20:28:35.636068 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.636041    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b325b4_c286_465f_83c9_623e0d441d17.slice/crio-52c7947ad1b050cd7270bcf7dd5c79119339283eb750f1b0bb567ea8f4bcb780 WatchSource:0}: Error finding container 52c7947ad1b050cd7270bcf7dd5c79119339283eb750f1b0bb567ea8f4bcb780: Status 404 returned error can't find the container with id 52c7947ad1b050cd7270bcf7dd5c79119339283eb750f1b0bb567ea8f4bcb780
Nov 03 20:28:35.639520 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.639469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:35.639520 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.639499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:35.639672 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.639634    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:35.639672 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.639651    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567984ddc5-9dl9n: secret "image-registry-tls" not found
Nov 03 20:28:35.639761 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.639679    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 20:28:35.639761 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.639707    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls podName:33bbc6dc-efa5-41c9-a6c6-d3a37542ea92 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.639687729 +0000 UTC m=+33.909645466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls") pod "image-registry-567984ddc5-9dl9n" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92") : secret "image-registry-tls" not found
Nov 03 20:28:35.639761 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.639723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:35.639910 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.639778    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls podName:68f3b7ee-a558-4c4d-8625-11f72d13b257 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.639739048 +0000 UTC m=+33.909696772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zd5zj" (UID: "68f3b7ee-a558-4c4d-8625-11f72d13b257") : secret "samples-operator-tls" not found
Nov 03 20:28:35.639910 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.639803    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:35.639910 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.639814    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd46f478-8pr5x: secret "image-registry-tls" not found
Nov 03 20:28:35.639910 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.639849    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls podName:a7c1d3a1-ac62-4cce-950a-640c97725c08 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.639836555 +0000 UTC m=+33.909794296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls") pod "image-registry-fbd46f478-8pr5x" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08") : secret "image-registry-tls" not found
Nov 03 20:28:35.648035 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.647874    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hjc54"]
Nov 03 20:28:35.651209 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.651155    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v455q"]
Nov 03 20:28:35.654192 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.654163    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c418d8_0eb1_44a4_9c0f_59d3d7d724af.slice/crio-261cff70c841b87369b2156d17642a305c347764a822b98bb5648467d24ab81a WatchSource:0}: Error finding container 261cff70c841b87369b2156d17642a305c347764a822b98bb5648467d24ab81a: Status 404 returned error can't find the container with id 261cff70c841b87369b2156d17642a305c347764a822b98bb5648467d24ab81a
Nov 03 20:28:35.655553 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.655399    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170428c7_02e4_40f5_a88c_1634d8b58b5a.slice/crio-a70653862c65783018189dfc9eaeac628cd2d7eaf59223862514c793002c953e WatchSource:0}: Error finding container a70653862c65783018189dfc9eaeac628cd2d7eaf59223862514c793002c953e: Status 404 returned error can't find the container with id a70653862c65783018189dfc9eaeac628cd2d7eaf59223862514c793002c953e
Nov 03 20:28:35.669157 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.669131    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"]
Nov 03 20:28:35.676969 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.676936    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859d83db_cfcc_459f_8ed2_8ec6a7e88b1a.slice/crio-9609b0c919b7fb3ea6f60ceb0da38f0315b03ca8f64f8b8227babc52c8b8df60 WatchSource:0}: Error finding container 9609b0c919b7fb3ea6f60ceb0da38f0315b03ca8f64f8b8227babc52c8b8df60: Status 404 returned error can't find the container with id 9609b0c919b7fb3ea6f60ceb0da38f0315b03ca8f64f8b8227babc52c8b8df60
Nov 03 20:28:35.713774 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.713730    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hpchn"]
Nov 03 20:28:35.718359 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.718336    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdf39cf_ae26_400e_b929_80512e786f5a.slice/crio-87a23b206956969cdaf1fb7169aed87ea8b90f8dbb513cf357704e9393654eda WatchSource:0}: Error finding container 87a23b206956969cdaf1fb7169aed87ea8b90f8dbb513cf357704e9393654eda: Status 404 returned error can't find the container with id 87a23b206956969cdaf1fb7169aed87ea8b90f8dbb513cf357704e9393654eda
Nov 03 20:28:35.729563 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.729539    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"]
Nov 03 20:28:35.733293 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:35.733268    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17826854_67bc_4852_97c1_83ebe2c648f7.slice/crio-6d34b35a471354374ecdafb1189d9ef8ffd3fe4417465c91072d4b805f10eb4e WatchSource:0}: Error finding container 6d34b35a471354374ecdafb1189d9ef8ffd3fe4417465c91072d4b805f10eb4e: Status 404 returned error can't find the container with id 6d34b35a471354374ecdafb1189d9ef8ffd3fe4417465c91072d4b805f10eb4e
Nov 03 20:28:35.740700 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:35.740677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:35.740823 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.740810    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:28:35.740877 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:35.740859    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls podName:545aca01-0ea8-4df2-b3bc-fa60f6f1c725 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:36.740845679 +0000 UTC m=+34.010803400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls") pod "dns-default-k9c8l" (UID: "545aca01-0ea8-4df2-b3bc-fa60f6f1c725") : secret "dns-default-metrics-tls" not found
Nov 03 20:28:36.005776 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.005690    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:36.203150 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.203116    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:36.245271 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.245240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:36.245424 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.245284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:36.245424 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.245349    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:36.245424 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.245407    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret podName:428ef791-c562-4786-b095-7c3b1113bfc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:08.24539364 +0000 UTC m=+65.515351362 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret") pod "global-pull-secret-syncer-8w5qd" (UID: "428ef791-c562-4786-b095-7c3b1113bfc8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:36.245424 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.245416    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:36.245561 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.245467    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs podName:e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:08.245452624 +0000 UTC m=+65.515410366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs") pod "network-metrics-daemon-g67qg" (UID: "e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:36.346552 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.346519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:28:36.346717 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.346573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:36.346717 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.346572    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:28:36.346839 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.346721    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:36.349448 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.349429    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:36.399528 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.399481    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:28:36.399695 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.399603    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kpkvl\""
Nov 03 20:28:36.399695 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.399619    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:28:36.399695 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.399645    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h9ssl\""
Nov 03 20:28:36.399820 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.399704    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jxjwz\""
Nov 03 20:28:36.414520 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.414493    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:36.549428 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.549394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.549526    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.549547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.549585    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert podName:3946b8e3-670c-407b-ba35-01cd37890b2d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:38.549569768 +0000 UTC m=+35.819527487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q6fg" (UID: "3946b8e3-670c-407b-ba35-01cd37890b2d") : secret "networking-console-plugin-cert" not found
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.549627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.549637    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:38.549623928 +0000 UTC m=+35.819581665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : configmap references non-existent config key: service-ca.crt
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.549672    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.549705    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls podName:a3b4704b-2627-4530-8100-838f2d8b7d31 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:38.549697742 +0000 UTC m=+35.819655462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4g2kg" (UID: "a3b4704b-2627-4530-8100-838f2d8b7d31") : secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.549718    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.549771    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 20:28:36.549922 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.549805    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:38.549796869 +0000 UTC m=+35.819754627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : secret "router-metrics-certs-default" not found
Nov 03 20:28:36.577703 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.577671    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qpwf2" event={"ID":"a533118b-20c1-478a-b7e7-42dd7b4f200c","Type":"ContainerStarted","Data":"a74206fc8958f8e73db1caa49b6dab01814e211a9001c08c9f7cf8ba72426eb4"}
Nov 03 20:28:36.578869 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.578831    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76" event={"ID":"17826854-67bc-4852-97c1-83ebe2c648f7","Type":"ContainerStarted","Data":"6d34b35a471354374ecdafb1189d9ef8ffd3fe4417465c91072d4b805f10eb4e"}
Nov 03 20:28:36.579984 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.579962    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" event={"ID":"170428c7-02e4-40f5-a88c-1634d8b58b5a","Type":"ContainerStarted","Data":"a70653862c65783018189dfc9eaeac628cd2d7eaf59223862514c793002c953e"}
Nov 03 20:28:36.580990 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.580953    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf" event={"ID":"e8b325b4-c286-465f-83c9-623e0d441d17","Type":"ContainerStarted","Data":"52c7947ad1b050cd7270bcf7dd5c79119339283eb750f1b0bb567ea8f4bcb780"}
Nov 03 20:28:36.582389 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.582369    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hjc54" event={"ID":"68c418d8-0eb1-44a4-9c0f-59d3d7d724af","Type":"ContainerStarted","Data":"a4747da8e4d94455d656619b125ee71508525ed0964be3827e282f24e1ecf4d9"}
Nov 03 20:28:36.582482 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.582401    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hjc54" event={"ID":"68c418d8-0eb1-44a4-9c0f-59d3d7d724af","Type":"ContainerStarted","Data":"261cff70c841b87369b2156d17642a305c347764a822b98bb5648467d24ab81a"}
Nov 03 20:28:36.586338 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.586308    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6vt8w" event={"ID":"1d9f21de-9ec8-48b7-aa66-5945a79d95e4","Type":"ContainerStarted","Data":"f8230d9807fed53979d3681654be4cf4fccbac1d4e11d298978704863e4cd343"}
Nov 03 20:28:36.587458 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.587439    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9" event={"ID":"9f9c06cb-46f7-440b-b395-a4c4e9a597cd","Type":"ContainerStarted","Data":"04eba560ee1d9cc80f006d14e5af55b65c817d405a813190c4f6f432e63186d4"}
Nov 03 20:28:36.588570 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.588549    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hpchn" event={"ID":"bfdf39cf-ae26-400e-b929-80512e786f5a","Type":"ContainerStarted","Data":"87a23b206956969cdaf1fb7169aed87ea8b90f8dbb513cf357704e9393654eda"}
Nov 03 20:28:36.589666 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.589645    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf" event={"ID":"859d83db-cfcc-459f-8ed2-8ec6a7e88b1a","Type":"ContainerStarted","Data":"9609b0c919b7fb3ea6f60ceb0da38f0315b03ca8f64f8b8227babc52c8b8df60"}
Nov 03 20:28:36.590703 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.590686    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-g2rqz" event={"ID":"151b0e6d-c5a6-469c-8d77-f7507ec6796a","Type":"ContainerStarted","Data":"37c190292dca8dd29400317822932d7169a10879152a603eab7442e65203d8c3"}
Nov 03 20:28:36.651071 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.650993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:36.651071 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.651036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:36.651325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.651151    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 20:28:36.651325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.651195    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:36.651325 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.651206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:36.651325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.651213    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd46f478-8pr5x: secret "image-registry-tls" not found
Nov 03 20:28:36.651325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.651232    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls podName:68f3b7ee-a558-4c4d-8625-11f72d13b257 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:38.651212479 +0000 UTC m=+35.921170213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zd5zj" (UID: "68f3b7ee-a558-4c4d-8625-11f72d13b257") : secret "samples-operator-tls" not found
Nov 03 20:28:36.651325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.651254    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls podName:a7c1d3a1-ac62-4cce-950a-640c97725c08 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:38.651246055 +0000 UTC m=+35.921203776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls") pod "image-registry-fbd46f478-8pr5x" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08") : secret "image-registry-tls" not found
Nov 03 20:28:36.651325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.651279    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:36.651325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.651295    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567984ddc5-9dl9n: secret "image-registry-tls" not found
Nov 03 20:28:36.651583 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.651342    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls podName:33bbc6dc-efa5-41c9-a6c6-d3a37542ea92 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:38.65132836 +0000 UTC m=+35.921286080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls") pod "image-registry-567984ddc5-9dl9n" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92") : secret "image-registry-tls" not found
Nov 03 20:28:36.752581 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.752549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:36.752791 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.752712    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:28:36.752867 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:36.752786    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls podName:545aca01-0ea8-4df2-b3bc-fa60f6f1c725 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:38.752767377 +0000 UTC m=+36.022725111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls") pod "dns-default-k9c8l" (UID: "545aca01-0ea8-4df2-b3bc-fa60f6f1c725") : secret "dns-default-metrics-tls" not found
Nov 03 20:28:36.771822 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.771763    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6vt8w" podStartSLOduration=464.771743233 podStartE2EDuration="7m44.771743233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:36.770722965 +0000 UTC m=+34.040680706" watchObservedRunningTime="2026-11-03 20:28:36.771743233 +0000 UTC m=+34.041700978"
Nov 03 20:28:36.789830 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.789794    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tjt8g"]
Nov 03 20:28:36.793541 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:28:36.793493    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1113fe81_67a7_4c02_ae59_d9fd15e22bd0.slice/crio-4189997ef87adf82a22a2bf89dcfae62b830dff5cf02b1c47ff8d0dd37368e3d WatchSource:0}: Error finding container 4189997ef87adf82a22a2bf89dcfae62b830dff5cf02b1c47ff8d0dd37368e3d: Status 404 returned error can't find the container with id 4189997ef87adf82a22a2bf89dcfae62b830dff5cf02b1c47ff8d0dd37368e3d
Nov 03 20:28:36.885055 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:36.885006    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hjc54" podStartSLOduration=464.884991153 podStartE2EDuration="7m44.884991153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:36.884316936 +0000 UTC m=+34.154274675" watchObservedRunningTime="2026-11-03 20:28:36.884991153 +0000 UTC m=+34.154948893"
Nov 03 20:28:37.599883 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:37.599808    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tjt8g" event={"ID":"1113fe81-67a7-4c02-ae59-d9fd15e22bd0","Type":"ContainerStarted","Data":"4189997ef87adf82a22a2bf89dcfae62b830dff5cf02b1c47ff8d0dd37368e3d"}
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.585425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.585590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.585628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.585677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.585866    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.585970    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls podName:a3b4704b-2627-4530-8100-838f2d8b7d31 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.585950652 +0000 UTC m=+39.855908387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4g2kg" (UID: "a3b4704b-2627-4530-8100-838f2d8b7d31") : secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.586463    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.586522    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.586505956 +0000 UTC m=+39.856463698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : secret "router-metrics-certs-default" not found
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.586589    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.586628    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert podName:3946b8e3-670c-407b-ba35-01cd37890b2d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.586617318 +0000 UTC m=+39.856575038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q6fg" (UID: "3946b8e3-670c-407b-ba35-01cd37890b2d") : secret "networking-console-plugin-cert" not found
Nov 03 20:28:38.587466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.586714    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.586698075 +0000 UTC m=+39.856655809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : configmap references non-existent config key: service-ca.crt
Nov 03 20:28:38.606636 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.605962    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf" event={"ID":"859d83db-cfcc-459f-8ed2-8ec6a7e88b1a","Type":"ContainerStarted","Data":"27b9b13196cb80a8b74d4ae94529feb9c54e57299654329a4a8a7895cf01400c"}
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.686870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.686926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.686956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.687136    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.687152    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd46f478-8pr5x: secret "image-registry-tls" not found
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.687210    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls podName:a7c1d3a1-ac62-4cce-950a-640c97725c08 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.687190453 +0000 UTC m=+39.957148189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls") pod "image-registry-fbd46f478-8pr5x" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08") : secret "image-registry-tls" not found
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.687282    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.687292    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567984ddc5-9dl9n: secret "image-registry-tls" not found
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.687326    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls podName:33bbc6dc-efa5-41c9-a6c6-d3a37542ea92 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.687314211 +0000 UTC m=+39.957271945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls") pod "image-registry-567984ddc5-9dl9n" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92") : secret "image-registry-tls" not found
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.687391    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 20:28:38.687452 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.687423    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls podName:68f3b7ee-a558-4c4d-8625-11f72d13b257 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.687412771 +0000 UTC m=+39.957370504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zd5zj" (UID: "68f3b7ee-a558-4c4d-8625-11f72d13b257") : secret "samples-operator-tls" not found
Nov 03 20:28:38.722001 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.720457    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qpwf2" podStartSLOduration=466.720444736 podStartE2EDuration="7m46.720444736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:37.001266234 +0000 UTC m=+34.271223982" watchObservedRunningTime="2026-11-03 20:28:38.720444736 +0000 UTC m=+35.990402477"
Nov 03 20:28:38.722001 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.720682    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf" podStartSLOduration=438.072461197 podStartE2EDuration="7m20.720675406s" totalImagesPullingTime="2.648214209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:38.719809611 +0000 UTC m=+35.989767351" watchObservedRunningTime="2026-11-03 20:28:38.720675406 +0000 UTC m=+35.990633150"
Nov 03 20:28:38.788775 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.787829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:38.788775 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.788337    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:28:38.788775 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:38.788407    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls podName:545aca01-0ea8-4df2-b3bc-fa60f6f1c725 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.788389084 +0000 UTC m=+40.058346808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls") pod "dns-default-k9c8l" (UID: "545aca01-0ea8-4df2-b3bc-fa60f6f1c725") : secret "dns-default-metrics-tls" not found
Nov 03 20:28:38.849084 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:38.848922    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:39.002533 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:39.002492    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:40.615406 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:40.615371    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/0.log"
Nov 03 20:28:40.615833 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:40.615418    2575 generic.go:356] "Generic (PLEG): container finished" podID="170428c7-02e4-40f5-a88c-1634d8b58b5a" containerID="1a52441858bf9f907691fdad44d78e494a48e002bdffe45f5c9f3b6e825abed1" exitCode=255
Nov 03 20:28:40.615833 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:40.615509    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" event={"ID":"170428c7-02e4-40f5-a88c-1634d8b58b5a","Type":"ContainerDied","Data":"1a52441858bf9f907691fdad44d78e494a48e002bdffe45f5c9f3b6e825abed1"}
Nov 03 20:28:40.615946 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:40.615880    2575 scope.go:122] "RemoveContainer" containerID="1a52441858bf9f907691fdad44d78e494a48e002bdffe45f5c9f3b6e825abed1"
Nov 03 20:28:40.789071 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:40.789037    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:40.946127 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:40.946025    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:41.567304 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.566381    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:41.622514 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.622488    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/1.log"
Nov 03 20:28:41.622994 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.622976    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/0.log"
Nov 03 20:28:41.623058 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.623010    2575 generic.go:356] "Generic (PLEG): container finished" podID="170428c7-02e4-40f5-a88c-1634d8b58b5a" containerID="68fb3fd547e043472b3c9043cf9b6a6764111438038ba1cd0351f9ebddb5a82c" exitCode=255
Nov 03 20:28:41.623141 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.623118    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" event={"ID":"170428c7-02e4-40f5-a88c-1634d8b58b5a","Type":"ContainerDied","Data":"68fb3fd547e043472b3c9043cf9b6a6764111438038ba1cd0351f9ebddb5a82c"}
Nov 03 20:28:41.623197 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.623184    2575 scope.go:122] "RemoveContainer" containerID="1a52441858bf9f907691fdad44d78e494a48e002bdffe45f5c9f3b6e825abed1"
Nov 03 20:28:41.623421 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.623404    2575 scope.go:122] "RemoveContainer" containerID="68fb3fd547e043472b3c9043cf9b6a6764111438038ba1cd0351f9ebddb5a82c"
Nov 03 20:28:41.623689 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:41.623665    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v455q_openshift-console-operator(170428c7-02e4-40f5-a88c-1634d8b58b5a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" podUID="170428c7-02e4-40f5-a88c-1634d8b58b5a"
Nov 03 20:28:41.625615 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.625593    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-g2rqz" event={"ID":"151b0e6d-c5a6-469c-8d77-f7507ec6796a","Type":"ContainerStarted","Data":"c6c55f51279080c35dc5b48f87e81e9104827a28872283bfeea7e6ec3fc1e5e1"}
Nov 03 20:28:41.902817 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:41.902768    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-g2rqz" podStartSLOduration=434.261387016 podStartE2EDuration="7m19.902755758s" totalImagesPullingTime="5.641368742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:41.852181178 +0000 UTC m=+39.122138918" watchObservedRunningTime="2026-11-03 20:28:41.902755758 +0000 UTC m=+39.172713498"
Nov 03 20:28:42.378519 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.378479    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lc4mt"]
Nov 03 20:28:42.394118 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.394076    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.423446 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.423405    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.423446 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.423446    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nx5\" (UniqueName: \"kubernetes.io/projected/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-api-access-46nx5\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.423656 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.423561    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca8995-d9ba-42fb-94b6-d7cf18696cce-data-volume\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.423656 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.423598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.423656 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.423624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46ca8995-d9ba-42fb-94b6-d7cf18696cce-crio-socket\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.488409 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.488381    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lc4mt"]
Nov 03 20:28:42.524228 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.524192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.524365 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.524251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46ca8995-d9ba-42fb-94b6-d7cf18696cce-crio-socket\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.524443 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.524394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.524443 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.524423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nx5\" (UniqueName: \"kubernetes.io/projected/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-api-access-46nx5\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.524540 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.524525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca8995-d9ba-42fb-94b6-d7cf18696cce-data-volume\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.524880 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.524846    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca8995-d9ba-42fb-94b6-d7cf18696cce-data-volume\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.525387 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.525365    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46ca8995-d9ba-42fb-94b6-d7cf18696cce-crio-socket\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.560847 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.560817    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:28:42.561015 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.560934    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntq9n\""
Nov 03 20:28:42.561015 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.561007    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:28:42.566243 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.566214    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:42.566381 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.566302    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:43.066281355 +0000 UTC m=+40.336239088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:42.566381 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.566327    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 20:28:42.566381 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.566360    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:43.066349358 +0000 UTC m=+40.336307090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : configmap "kube-rbac-proxy" not found
Nov 03 20:28:42.583084 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.583056    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nx5\" (UniqueName: \"kubernetes.io/projected/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-api-access-46nx5\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:42.612609 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.612572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.625738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.625784    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.625816    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.625882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.625995    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.626040    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.626027517 +0000 UTC m=+47.895985248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : secret "router-metrics-certs-default" not found
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.626073    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.626054253 +0000 UTC m=+47.896011978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : configmap references non-existent config key: service-ca.crt
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.626117    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.626141    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 20:28:42.626185 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.626161    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls podName:a3b4704b-2627-4530-8100-838f2d8b7d31 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.626149792 +0000 UTC m=+47.896107530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4g2kg" (UID: "a3b4704b-2627-4530-8100-838f2d8b7d31") : secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:42.626923 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.626224    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert podName:3946b8e3-670c-407b-ba35-01cd37890b2d nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.626206127 +0000 UTC m=+47.896163861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q6fg" (UID: "3946b8e3-670c-407b-ba35-01cd37890b2d") : secret "networking-console-plugin-cert" not found
Nov 03 20:28:42.631683 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.631656    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9" event={"ID":"9f9c06cb-46f7-440b-b395-a4c4e9a597cd","Type":"ContainerStarted","Data":"e12f02b204568606b23b8b1c571653794a5d0f6e278ea7c1a5a298f2cedba7a2"}
Nov 03 20:28:42.634193 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.634115    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tjt8g" event={"ID":"1113fe81-67a7-4c02-ae59-d9fd15e22bd0","Type":"ContainerStarted","Data":"0c84645f2d5a9d69b687a4ac3bec0074f9718f093910ccc0486e7b000fffb0eb"}
Nov 03 20:28:42.634193 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.634164    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:28:42.636324 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.636293    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76" event={"ID":"17826854-67bc-4852-97c1-83ebe2c648f7","Type":"ContainerStarted","Data":"c857420e28537470ed0438c7812f806bdbf97e9cf27d29d422dcd925d76be0e2"}
Nov 03 20:28:42.638147 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.638130    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/1.log"
Nov 03 20:28:42.638545 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.638525    2575 scope.go:122] "RemoveContainer" containerID="68fb3fd547e043472b3c9043cf9b6a6764111438038ba1cd0351f9ebddb5a82c"
Nov 03 20:28:42.638717 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.638697    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v455q_openshift-console-operator(170428c7-02e4-40f5-a88c-1634d8b58b5a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" podUID="170428c7-02e4-40f5-a88c-1634d8b58b5a"
Nov 03 20:28:42.727439 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.727399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:42.727606 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.727483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:42.727606 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.727514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:42.728140 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.728116    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 20:28:42.728243 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.728180    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls podName:68f3b7ee-a558-4c4d-8625-11f72d13b257 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.728160713 +0000 UTC m=+47.998118462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zd5zj" (UID: "68f3b7ee-a558-4c4d-8625-11f72d13b257") : secret "samples-operator-tls" not found
Nov 03 20:28:42.728427 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.728406    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:42.728493 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.728432    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567984ddc5-9dl9n: secret "image-registry-tls" not found
Nov 03 20:28:42.728493 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.728479    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls podName:33bbc6dc-efa5-41c9-a6c6-d3a37542ea92 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.728462165 +0000 UTC m=+47.998419900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls") pod "image-registry-567984ddc5-9dl9n" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92") : secret "image-registry-tls" not found
Nov 03 20:28:42.728674 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.728657    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:42.728674 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.728674    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd46f478-8pr5x: secret "image-registry-tls" not found
Nov 03 20:28:42.728760 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.728706    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls podName:a7c1d3a1-ac62-4cce-950a-640c97725c08 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.728696199 +0000 UTC m=+47.998653918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls") pod "image-registry-fbd46f478-8pr5x" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08") : secret "image-registry-tls" not found
Nov 03 20:28:42.745944 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.745875    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9" podStartSLOduration=436.615350279 podStartE2EDuration="7m22.745857692s" totalImagesPullingTime="6.130507413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:42.745809231 +0000 UTC m=+40.015766975" watchObservedRunningTime="2026-11-03 20:28:42.745857692 +0000 UTC m=+40.015815435"
Nov 03 20:28:42.776218 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.776047    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:42.828584 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.828539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:42.828767 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.828692    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:28:42.828767 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:42.828763    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls podName:545aca01-0ea8-4df2-b3bc-fa60f6f1c725 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.82874071 +0000 UTC m=+48.098698448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls") pod "dns-default-k9c8l" (UID: "545aca01-0ea8-4df2-b3bc-fa60f6f1c725") : secret "dns-default-metrics-tls" not found
Nov 03 20:28:42.861422 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:42.861348    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tjt8g" podStartSLOduration=33.397047327 podStartE2EDuration="38.861329713s" totalImagesPullingTime="5.464282386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:42.860673362 +0000 UTC m=+40.130631105" watchObservedRunningTime="2026-11-03 20:28:42.861329713 +0000 UTC m=+40.131287455"
Nov 03 20:28:43.130763 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.130693    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76" podStartSLOduration=432.454118211 podStartE2EDuration="7m19.13067064s" totalImagesPullingTime="6.676552429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:42.973748344 +0000 UTC m=+40.243706086" watchObservedRunningTime="2026-11-03 20:28:43.13067064 +0000 UTC m=+40.400628380"
Nov 03 20:28:43.131203 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.131181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:43.131323 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.131309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:43.131387 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:43.131349    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:43.131432 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:43.131414    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 20:28:43.131466 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:43.131417    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.131398863 +0000 UTC m=+41.401356591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:43.131514 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:43.131471    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.13146 +0000 UTC m=+41.401417719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : configmap "kube-rbac-proxy" not found
Nov 03 20:28:43.221449 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.221404    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:43.370276 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.370242    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:43.522059 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.522027    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:43.646369 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.646325    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf" event={"ID":"e8b325b4-c286-465f-83c9-623e0d441d17","Type":"ContainerStarted","Data":"4c6264356a8bcdc67869ccc397c7f8f97d398d6adb84872ef592d156d84170c3"}
Nov 03 20:28:43.668729 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.668696    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:43.770408 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.770306    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf" podStartSLOduration=431.938578432 podStartE2EDuration="7m19.770286481s" totalImagesPullingTime="7.831708049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:43.76969173 +0000 UTC m=+41.039649472" watchObservedRunningTime="2026-11-03 20:28:43.770286481 +0000 UTC m=+41.040244221"
Nov 03 20:28:43.826593 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:43.826557    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:44.142926 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:44.142887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:44.143137 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:44.143022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:44.143137 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:44.143069    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:44.143245 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:44.143145    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 20:28:44.143245 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:44.143165    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:46.143144441 +0000 UTC m=+43.413102160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:44.143245 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:44.143208    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:46.143190918 +0000 UTC m=+43.413148656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : configmap "kube-rbac-proxy" not found
Nov 03 20:28:44.620521 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:44.620459    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:44.651612 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:44.651569    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hpchn" event={"ID":"bfdf39cf-ae26-400e-b929-80512e786f5a","Type":"ContainerStarted","Data":"a99f2b55da3e55e2e1c4b031a763b8a28b680eacf1dfa58900b10400d5047ae5"}
Nov 03 20:28:44.766602 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:44.766546    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hpchn" podStartSLOduration=2.189538488 podStartE2EDuration="10.766530824s" totalImagesPullingTime="8.576992336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:44.765375499 +0000 UTC m=+42.035333241" watchObservedRunningTime="2026-11-03 20:28:44.766530824 +0000 UTC m=+42.036488583"
Nov 03 20:28:44.773424 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:44.773386    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:45.104617 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:45.104519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:45.223913 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:45.223881    2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:45.224065 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:45.223939    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:45.224334 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:45.224321    2575 scope.go:122] "RemoveContainer" containerID="68fb3fd547e043472b3c9043cf9b6a6764111438038ba1cd0351f9ebddb5a82c"
Nov 03 20:28:45.224539 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:45.224519    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v455q_openshift-console-operator(170428c7-02e4-40f5-a88c-1634d8b58b5a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" podUID="170428c7-02e4-40f5-a88c-1634d8b58b5a"
Nov 03 20:28:45.265371 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:45.265306    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:46.162962 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:46.162926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:46.163383 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:46.163050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:46.163383 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:46.163055    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 20:28:46.163383 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:46.163139    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.163121 +0000 UTC m=+47.433078740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : configmap "kube-rbac-proxy" not found
Nov 03 20:28:46.163383 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:46.163190    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:46.163383 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:46.163238    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:50.163226732 +0000 UTC m=+47.433184452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:46.348624 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:46.348589    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:46.510930 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:46.510900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:46.917199 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:46.917162    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:47.535317 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:47.535281    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:49.855071 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:49.855035    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:50.202505 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.202406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:50.202677 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.202535    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:50.202677 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.202559    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:50.202677 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.202603    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:28:58.202584982 +0000 UTC m=+55.472542718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:50.203066 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.203046    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:50.707786 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.707740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:50.707988 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.707870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:28:50.707988 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.707910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:28:50.707988 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.707915    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 20:28:50.707988 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.707949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:28:50.707988 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.707981    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:06.707962129 +0000 UTC m=+63.977919866 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : secret "router-metrics-certs-default" not found
Nov 03 20:28:50.708255 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.708006    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 20:28:50.708255 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.708021    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:50.708255 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.708056    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls podName:a3b4704b-2627-4530-8100-838f2d8b7d31 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:06.708046097 +0000 UTC m=+63.978003835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4g2kg" (UID: "a3b4704b-2627-4530-8100-838f2d8b7d31") : secret "cluster-monitoring-operator-tls" not found
Nov 03 20:28:50.708255 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.708072    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert podName:3946b8e3-670c-407b-ba35-01cd37890b2d nodeName:}" failed. No retries permitted until 2026-11-03 20:29:06.708063778 +0000 UTC m=+63.978021500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q6fg" (UID: "3946b8e3-670c-407b-ba35-01cd37890b2d") : secret "networking-console-plugin-cert" not found
Nov 03 20:28:50.708255 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.708180    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:06.70816011 +0000 UTC m=+63.978117831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : configmap references non-existent config key: service-ca.crt
Nov 03 20:28:50.808646 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.808607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:28:50.808839 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.808655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:28:50.808839 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.808758    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 20:28:50.808839 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.808766    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:50.808839 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.808792    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567984ddc5-9dl9n: secret "image-registry-tls" not found
Nov 03 20:28:50.808839 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.808794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:28:50.808839 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.808818    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls podName:68f3b7ee-a558-4c4d-8625-11f72d13b257 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:06.808802594 +0000 UTC m=+64.078760329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zd5zj" (UID: "68f3b7ee-a558-4c4d-8625-11f72d13b257") : secret "samples-operator-tls" not found
Nov 03 20:28:50.809044 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.808846    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls podName:33bbc6dc-efa5-41c9-a6c6-d3a37542ea92 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:06.808827762 +0000 UTC m=+64.078785494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls") pod "image-registry-567984ddc5-9dl9n" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92") : secret "image-registry-tls" not found
Nov 03 20:28:50.809044 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.808865    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:28:50.809044 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.808877    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd46f478-8pr5x: secret "image-registry-tls" not found
Nov 03 20:28:50.809044 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.808949    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls podName:a7c1d3a1-ac62-4cce-950a-640c97725c08 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:06.808935196 +0000 UTC m=+64.078892932 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls") pod "image-registry-fbd46f478-8pr5x" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08") : secret "image-registry-tls" not found
Nov 03 20:28:50.909553 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:50.909520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:28:50.909890 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.909684    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:28:50.909890 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:50.909757    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls podName:545aca01-0ea8-4df2-b3bc-fa60f6f1c725 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:06.909741265 +0000 UTC m=+64.179698987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls") pod "dns-default-k9c8l" (UID: "545aca01-0ea8-4df2-b3bc-fa60f6f1c725") : secret "dns-default-metrics-tls" not found
Nov 03 20:28:51.216158 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:51.216129    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-np7pp_53ee9fcc-846d-4ca8-835f-58d77e3f1a63/dns-node-resolver/0.log"
Nov 03 20:28:52.215902 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.215870    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nlnjm_9d8dc680-3f86-4ec8-949a-18cb41c680d1/node-ca/0.log"
Nov 03 20:28:52.346567 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.346532    2575 scope.go:122] "RemoveContainer" containerID="68fb3fd547e043472b3c9043cf9b6a6764111438038ba1cd0351f9ebddb5a82c"
Nov 03 20:28:52.678332 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.678307    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/2.log"
Nov 03 20:28:52.678689 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.678674    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/1.log"
Nov 03 20:28:52.678733 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.678708    2575 generic.go:356] "Generic (PLEG): container finished" podID="170428c7-02e4-40f5-a88c-1634d8b58b5a" containerID="2362ef8b90f49540f0b568ae8ec045a9f94eb13227b076deec426f29acaad4e3" exitCode=255
Nov 03 20:28:52.678767 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.678740    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" event={"ID":"170428c7-02e4-40f5-a88c-1634d8b58b5a","Type":"ContainerDied","Data":"2362ef8b90f49540f0b568ae8ec045a9f94eb13227b076deec426f29acaad4e3"}
Nov 03 20:28:52.678796 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.678786    2575 scope.go:122] "RemoveContainer" containerID="68fb3fd547e043472b3c9043cf9b6a6764111438038ba1cd0351f9ebddb5a82c"
Nov 03 20:28:52.679172 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.679148    2575 scope.go:122] "RemoveContainer" containerID="2362ef8b90f49540f0b568ae8ec045a9f94eb13227b076deec426f29acaad4e3"
Nov 03 20:28:52.679397 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:52.679373    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v455q_openshift-console-operator(170428c7-02e4-40f5-a88c-1634d8b58b5a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" podUID="170428c7-02e4-40f5-a88c-1634d8b58b5a"
Nov 03 20:28:52.816790 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:52.816759    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hpchn_bfdf39cf-ae26-400e-b929-80512e786f5a/serve-healthcheck-canary/0.log"
Nov 03 20:28:53.192341 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:53.192311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:53.616811 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:53.616783    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-66vk9_9f9c06cb-46f7-440b-b395-a4c4e9a597cd/kube-storage-version-migrator-operator/0.log"
Nov 03 20:28:53.683872 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:53.683846    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/2.log"
Nov 03 20:28:55.223636 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:55.223599    2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:55.224020 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:55.223682    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:28:55.224020 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:55.223971    2575 scope.go:122] "RemoveContainer" containerID="2362ef8b90f49540f0b568ae8ec045a9f94eb13227b076deec426f29acaad4e3"
Nov 03 20:28:55.224195 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:55.224173    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v455q_openshift-console-operator(170428c7-02e4-40f5-a88c-1634d8b58b5a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" podUID="170428c7-02e4-40f5-a88c-1634d8b58b5a"
Nov 03 20:28:58.275760 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:58.275711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:28:58.276303 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:58.275872    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:58.276303 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:28:58.275936    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls podName:46ca8995-d9ba-42fb-94b6-d7cf18696cce nodeName:}" failed. No retries permitted until 2026-11-03 20:29:14.275922133 +0000 UTC m=+71.545879851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc4mt" (UID: "46ca8995-d9ba-42fb-94b6-d7cf18696cce") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:28:58.821580 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:58.821551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:58.980466 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:58.980429    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:59.569148 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:59.569121    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:28:59.621788 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:59.621757    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:59.775300 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:28:59.775267    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:06.759765 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:06.759724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:06.759818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:06.759841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:06.759868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.759869    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.759935    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:38.759916982 +0000 UTC m=+96.029874721 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : secret "router-metrics-certs-default" not found
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.759976    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.759980    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.759990    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle podName:ee248b83-7c43-46fe-809c-69b61ca6254a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:38.759978491 +0000 UTC m=+96.029936214 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle") pod "router-default-c7454997c-ccrh5" (UID: "ee248b83-7c43-46fe-809c-69b61ca6254a") : configmap references non-existent config key: service-ca.crt
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.760016    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls podName:a3b4704b-2627-4530-8100-838f2d8b7d31 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:38.760007427 +0000 UTC m=+96.029965151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4g2kg" (UID: "a3b4704b-2627-4530-8100-838f2d8b7d31") : secret "cluster-monitoring-operator-tls" not found
Nov 03 20:29:06.760325 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.760049    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert podName:3946b8e3-670c-407b-ba35-01cd37890b2d nodeName:}" failed. No retries permitted until 2026-11-03 20:29:38.760031751 +0000 UTC m=+96.029989486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6q6fg" (UID: "3946b8e3-670c-407b-ba35-01cd37890b2d") : secret "networking-console-plugin-cert" not found
Nov 03 20:29:06.861071 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:06.861029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") pod \"image-registry-567984ddc5-9dl9n\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") " pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:29:06.861071 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:06.861072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:06.861347 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:06.861107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") pod \"image-registry-fbd46f478-8pr5x\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") " pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:29:06.861347 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.861236    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:29:06.861347 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.861234    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 20:29:06.861347 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.861263    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 20:29:06.861347 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.861320    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls podName:68f3b7ee-a558-4c4d-8625-11f72d13b257 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:38.861306131 +0000 UTC m=+96.131263854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zd5zj" (UID: "68f3b7ee-a558-4c4d-8625-11f72d13b257") : secret "samples-operator-tls" not found
Nov 03 20:29:06.861347 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.861268    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567984ddc5-9dl9n: secret "image-registry-tls" not found
Nov 03 20:29:06.861530 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.861383    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls podName:33bbc6dc-efa5-41c9-a6c6-d3a37542ea92 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:38.861371236 +0000 UTC m=+96.131328955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls") pod "image-registry-567984ddc5-9dl9n" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92") : secret "image-registry-tls" not found
Nov 03 20:29:06.861530 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.861248    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd46f478-8pr5x: secret "image-registry-tls" not found
Nov 03 20:29:06.861530 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.861420    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls podName:a7c1d3a1-ac62-4cce-950a-640c97725c08 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:38.86141128 +0000 UTC m=+96.131369002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls") pod "image-registry-fbd46f478-8pr5x" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08") : secret "image-registry-tls" not found
Nov 03 20:29:06.961903 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:06.961865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:06.962115 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.962039    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:29:06.962184 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:06.962135    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls podName:545aca01-0ea8-4df2-b3bc-fa60f6f1c725 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:38.962116836 +0000 UTC m=+96.232074571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls") pod "dns-default-k9c8l" (UID: "545aca01-0ea8-4df2-b3bc-fa60f6f1c725") : secret "dns-default-metrics-tls" not found
Nov 03 20:29:07.036492 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:07.036422    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:07.450897 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:07.450867    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:07.599511 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:07.599471    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:07.769420 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:07.769334    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:08.272772 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.272721    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:08.272772 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.272777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:29:08.279429 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.279397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:08.324795 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.324768    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:29:08.324908 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.324847    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:29:08.333000 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:08.332978    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 20:29:08.333063 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:08.333052    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs podName:e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6 nodeName:}" failed. No retries permitted until 2026-11-03 20:30:12.333031217 +0000 UTC m=+129.602988937 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs") pod "network-metrics-daemon-g67qg" (UID: "e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6") : secret "metrics-daemon-secret" not found
Nov 03 20:29:08.335264 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.335239    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:08.532496 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.532419    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kpkvl\""
Nov 03 20:29:08.541517 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.541490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:08.909967 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.909937    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8w5qd"]
Nov 03 20:29:08.913448 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:29:08.913419    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428ef791_c562_4786_b095_7c3b1113bfc8.slice/crio-c5af1f37546b34aaf445847294f005402ecab75b88ba4aa4dff65991ab22c41b WatchSource:0}: Error finding container c5af1f37546b34aaf445847294f005402ecab75b88ba4aa4dff65991ab22c41b: Status 404 returned error can't find the container with id c5af1f37546b34aaf445847294f005402ecab75b88ba4aa4dff65991ab22c41b
Nov 03 20:29:08.925922 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:08.925894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:09.133464 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.133422    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:09.315390 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.315312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:09.531002 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.530925    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:09.548825 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.548791    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-567984ddc5-9dl9n"]
Nov 03 20:29:09.549109 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:09.549066    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-567984ddc5-9dl9n" podUID="33bbc6dc-efa5-41c9-a6c6-d3a37542ea92"
Nov 03 20:29:09.725277 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.725240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:09.734650 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.734629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:29:09.734784 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.734624    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8w5qd" event={"ID":"428ef791-c562-4786-b095-7c3b1113bfc8","Type":"ContainerStarted","Data":"c5af1f37546b34aaf445847294f005402ecab75b88ba4aa4dff65991ab22c41b"}
Nov 03 20:29:09.741272 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.741250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:29:09.875787 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.875752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:09.887710 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.887679    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-installation-pull-secrets\") pod \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") "
Nov 03 20:29:09.887864 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.887733    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-bound-sa-token\") pod \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") "
Nov 03 20:29:09.887864 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.887760    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-kube-api-access-5f7nw\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-kube-api-access-5f7nw\") pod \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") "
Nov 03 20:29:09.887864 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.887784    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-certificates\") pod \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") "
Nov 03 20:29:09.887864 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.887831    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-trusted-ca\") pod \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") "
Nov 03 20:29:09.887864 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.887855    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-image-registry-private-configuration\") pod \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") "
Nov 03 20:29:09.888141 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.887895    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-ca-trust-extracted\") pod \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\" (UID: \"33bbc6dc-efa5-41c9-a6c6-d3a37542ea92\") "
Nov 03 20:29:09.888411 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.888350    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-certificates" pod "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:09.888411 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.888365    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-ca-trust-extracted" pod "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:09.888559 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.888413    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-trusted-ca" pod "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:09.890337 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.890302    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-image-registry-private-configuration" pod "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:09.890576 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.890553    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-installation-pull-secrets" pod "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:09.890827 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.890809    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-kube-api-access-5f7nw" pod "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92"). InnerVolumeSpecName "kube-api-access-5f7nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:09.890903 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.890882    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-bound-sa-token" pod "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92" (UID: "33bbc6dc-efa5-41c9-a6c6-d3a37542ea92"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:09.993364 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.989952    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-bound-sa-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:09.993364 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.990044    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7nw\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-kube-api-access-5f7nw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:09.993364 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.990061    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-certificates\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:09.993364 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.990077    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-trusted-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:09.993364 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.990126    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-image-registry-private-configuration\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:09.993364 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.990141    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-ca-trust-extracted\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:09.993364 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:09.990159    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-installation-pull-secrets\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:10.032734 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:10.032696    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:10.737361 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:10.737327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567984ddc5-9dl9n"
Nov 03 20:29:10.927244 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:10.927214    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:11.066292 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.066206    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-567984ddc5-9dl9n"]
Nov 03 20:29:11.116494 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.116461    2575 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-567984ddc5-9dl9n"]
Nov 03 20:29:11.122270 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.122244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:11.201962 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.201927    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92-registry-tls\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:11.312770 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.312733    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:11.351812 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.351777    2575 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33bbc6dc-efa5-41c9-a6c6-d3a37542ea92" path="/var/lib/kubelet/pods/33bbc6dc-efa5-41c9-a6c6-d3a37542ea92/volumes"
Nov 03 20:29:11.520195 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.520161    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:11.775681 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.775600    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:11.815647 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.815610    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fbd46f478-8pr5x"]
Nov 03 20:29:11.815901 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:11.815869    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-fbd46f478-8pr5x" podUID="a7c1d3a1-ac62-4cce-950a-640c97725c08"
Nov 03 20:29:11.928663 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:11.928631    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:12.625774 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.625738    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:12.742970 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.742943    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:29:12.777609 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.777572    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:29:12.911622 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.911541    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:12.916909 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.916881    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-trusted-ca\") pod \"a7c1d3a1-ac62-4cce-950a-640c97725c08\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") "
Nov 03 20:29:12.917047 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.916935    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-certificates\") pod \"a7c1d3a1-ac62-4cce-950a-640c97725c08\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") "
Nov 03 20:29:12.917047 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.916978    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-installation-pull-secrets\") pod \"a7c1d3a1-ac62-4cce-950a-640c97725c08\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") "
Nov 03 20:29:12.917047 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.917029    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-image-registry-private-configuration\") pod \"a7c1d3a1-ac62-4cce-950a-640c97725c08\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") "
Nov 03 20:29:12.917210 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.917069    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-kube-api-access-rj2xz\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-kube-api-access-rj2xz\") pod \"a7c1d3a1-ac62-4cce-950a-640c97725c08\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") "
Nov 03 20:29:12.917210 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.917131    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-bound-sa-token\") pod \"a7c1d3a1-ac62-4cce-950a-640c97725c08\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") "
Nov 03 20:29:12.917210 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.917173    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c1d3a1-ac62-4cce-950a-640c97725c08-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7c1d3a1-ac62-4cce-950a-640c97725c08-ca-trust-extracted\") pod \"a7c1d3a1-ac62-4cce-950a-640c97725c08\" (UID: \"a7c1d3a1-ac62-4cce-950a-640c97725c08\") "
Nov 03 20:29:12.917394 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.917368    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-trusted-ca" pod "a7c1d3a1-ac62-4cce-950a-640c97725c08" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:12.917506 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.917491    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-trusted-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:12.917761 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.917716    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-certificates" pod "a7c1d3a1-ac62-4cce-950a-640c97725c08" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:12.917761 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.917727    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c1d3a1-ac62-4cce-950a-640c97725c08-ca-trust-extracted" pod "a7c1d3a1-ac62-4cce-950a-640c97725c08" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:12.919968 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.919941    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-bound-sa-token" pod "a7c1d3a1-ac62-4cce-950a-640c97725c08" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:12.920270 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.920239    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-kube-api-access-rj2xz" pod "a7c1d3a1-ac62-4cce-950a-640c97725c08" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08"). InnerVolumeSpecName "kube-api-access-rj2xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:12.920705 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.920681    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-image-registry-private-configuration" pod "a7c1d3a1-ac62-4cce-950a-640c97725c08" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:12.920937 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:12.920921    2575 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-installation-pull-secrets" pod "a7c1d3a1-ac62-4cce-950a-640c97725c08" (UID: "a7c1d3a1-ac62-4cce-950a-640c97725c08"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:13.018451 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.018413    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-certificates\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.018451 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.018439    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-installation-pull-secrets\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.018451 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.018450    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7c1d3a1-ac62-4cce-950a-640c97725c08-image-registry-private-configuration\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.018451 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.018460    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj2xz\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-kube-api-access-rj2xz\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.018729 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.018468    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-bound-sa-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.018729 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.018477    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7c1d3a1-ac62-4cce-950a-640c97725c08-ca-trust-extracted\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.118156 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.118118    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:13.259169 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.259062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:13.370968 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:13.370938    2575 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c1d3a1_ac62_4cce_950a_640c97725c08.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:29:13.403776 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.403747    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:13.552014 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.551937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:13.649458 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.649432    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:29:13.719859 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.719806    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:13.745474 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.745443    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbd46f478-8pr5x"
Nov 03 20:29:13.936215 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.936185    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fbd46f478-8pr5x"]
Nov 03 20:29:13.986729 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:13.986691    2575 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fbd46f478-8pr5x"]
Nov 03 20:29:14.131321 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.131284    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7c1d3a1-ac62-4cce-950a-640c97725c08-registry-tls\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:14.332470 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.332438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:14.334922 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.334899    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:14.346338 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.346312    2575 scope.go:122] "RemoveContainer" containerID="2362ef8b90f49540f0b568ae8ec045a9f94eb13227b076deec426f29acaad4e3"
Nov 03 20:29:14.616399 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.616323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:14.631839 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.631814    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntq9n\""
Nov 03 20:29:14.641113 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.641079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:14.750242 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.750212    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:29:14.750663 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.750648    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/2.log"
Nov 03 20:29:14.750705 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.750681    2575 generic.go:356] "Generic (PLEG): container finished" podID="170428c7-02e4-40f5-a88c-1634d8b58b5a" containerID="8921072839d44f3eea0077736b80615c3b142efc9f2f5d07700e5f58f72abfb2" exitCode=255
Nov 03 20:29:14.750734 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.750713    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" event={"ID":"170428c7-02e4-40f5-a88c-1634d8b58b5a","Type":"ContainerDied","Data":"8921072839d44f3eea0077736b80615c3b142efc9f2f5d07700e5f58f72abfb2"}
Nov 03 20:29:14.750766 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.750756    2575 scope.go:122] "RemoveContainer" containerID="2362ef8b90f49540f0b568ae8ec045a9f94eb13227b076deec426f29acaad4e3"
Nov 03 20:29:14.751141 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.751123    2575 scope.go:122] "RemoveContainer" containerID="8921072839d44f3eea0077736b80615c3b142efc9f2f5d07700e5f58f72abfb2"
Nov 03 20:29:14.751323 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:14.751304    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v455q_openshift-console-operator(170428c7-02e4-40f5-a88c-1634d8b58b5a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" podUID="170428c7-02e4-40f5-a88c-1634d8b58b5a"
Nov 03 20:29:14.775111 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.775056    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:14.922779 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:14.922664    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:15.025614 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.025582    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lc4mt"]
Nov 03 20:29:15.029318 ip-10-0-128-49 kubenswrapper[2575]: W1103 20:29:15.029294    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ca8995_d9ba_42fb_94b6_d7cf18696cce.slice/crio-3d09e80183bf9d2b72eecb5a3e06b8aff84d47ad81755d84c86e4798c5e83d0e WatchSource:0}: Error finding container 3d09e80183bf9d2b72eecb5a3e06b8aff84d47ad81755d84c86e4798c5e83d0e: Status 404 returned error can't find the container with id 3d09e80183bf9d2b72eecb5a3e06b8aff84d47ad81755d84c86e4798c5e83d0e
Nov 03 20:29:15.119185 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.119157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:15.223819 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.223735    2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:15.223819 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.223801    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:15.313205 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.313176    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:15.350480 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.350449    2575 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c1d3a1-ac62-4cce-950a-640c97725c08" path="/var/lib/kubelet/pods/a7c1d3a1-ac62-4cce-950a-640c97725c08/volumes"
Nov 03 20:29:15.519560 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.519490    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:15.727309 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.727276    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:15.756364 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.756326    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lc4mt" event={"ID":"46ca8995-d9ba-42fb-94b6-d7cf18696cce","Type":"ContainerStarted","Data":"b91d473a9d084580c622c624d2aab579356972ee01fd0658109ea1e504052a97"}
Nov 03 20:29:15.756849 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.756386    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lc4mt" event={"ID":"46ca8995-d9ba-42fb-94b6-d7cf18696cce","Type":"ContainerStarted","Data":"3d09e80183bf9d2b72eecb5a3e06b8aff84d47ad81755d84c86e4798c5e83d0e"}
Nov 03 20:29:15.758143 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.758107    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:29:15.758457 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:15.758440    2575 scope.go:122] "RemoveContainer" containerID="8921072839d44f3eea0077736b80615c3b142efc9f2f5d07700e5f58f72abfb2"
Nov 03 20:29:15.758622 ip-10-0-128-49 kubenswrapper[2575]: E1103 20:29:15.758603    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v455q_openshift-console-operator(170428c7-02e4-40f5-a88c-1634d8b58b5a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" podUID="170428c7-02e4-40f5-a88c-1634d8b58b5a"
Nov 03 20:29:16.917598 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:16.917572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.118742 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:17.118705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.312387 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:17.312317    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.517049 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:17.517011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.719793 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:17.719762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.766755 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:17.766726    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lc4mt" event={"ID":"46ca8995-d9ba-42fb-94b6-d7cf18696cce","Type":"ContainerStarted","Data":"056e2eec5f9d7bbc55d1b5d1621715046eb9be02c7651d775794bc60bfa46a64"}
Nov 03 20:29:19.775457 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:19.775424    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8w5qd" event={"ID":"428ef791-c562-4786-b095-7c3b1113bfc8","Type":"ContainerStarted","Data":"7cac2890459c73c27a1e1ac347e4ea4adca3ae9b8e9a6fb2d1389081668757d4"}
Nov 03 20:29:19.777798 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:19.777766    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lc4mt" event={"ID":"46ca8995-d9ba-42fb-94b6-d7cf18696cce","Type":"ContainerStarted","Data":"b8a74e1b443b52cdd8a86310869b717cbfb14b92cb75fa73a6549e00f3f6020a"}
Nov 03 20:29:19.889160 ip-10-0-128-49 kubenswrapper[2575]: I1103 20:29:19.889081    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8w5qd" podStartSLOduration=65.127172218 podStartE2EDuration="1m15.889065517s" totalImagesPullingTime="10.761893299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:19.888035097 +0000 UTC m=+77.157992840" watchObservedRunningTime="2026-11-03 20:29:19.889065517 +0000 UTC m=+77.159023258"
Nov 03 20:29:19.934422 ip-10-0-128-49 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 20:29:19.944742 ip-10-0-128-49 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 20:29:19.944915 ip-10-0-128-49 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 20:29:19.944989 ip-10-0-128-49 systemd[1]: kubelet.service: Consumed 4.557s CPU time.
Nov 03 20:29:19.963661 ip-10-0-128-49 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:29:20.026111 ip-10-0-128-49 kubenswrapper[5196]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:29:20.026111 ip-10-0-128-49 kubenswrapper[5196]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:29:20.026111 ip-10-0-128-49 kubenswrapper[5196]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:29:20.026111 ip-10-0-128-49 kubenswrapper[5196]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:29:20.028266 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028171    5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:20.028266 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028267    5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028272    5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028277    5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028280    5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028283    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028285    5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028288    5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028291    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028293    5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028296    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028298    5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028301    5196 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028304    5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028313    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028316    5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028318    5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028321    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028324    5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028327    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028330    5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:20.028338 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028333    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028336    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028339    5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028341    5196 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028345    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028347    5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028351    5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028354    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028356    5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028359    5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028361    5196 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028364    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028366    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028369    5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028371    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028374    5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028376    5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028379    5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028381    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:20.028850 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028384    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028386    5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028390    5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028392    5196 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028395    5196 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028397    5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028400    5196 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028403    5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028406    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028408    5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028410    5196 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028413    5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028415    5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028418    5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028421    5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028424    5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028427    5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028429    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028432    5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028434    5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:20.029381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028436    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028439    5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028443    5196 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028446    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028448    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028451    5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028453    5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028456    5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028459    5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028462    5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028464    5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028467    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028470    5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028473    5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028476    5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028479    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028482    5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028484    5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028487    5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028490    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:20.029888 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028492    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028495    5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028497    5196 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028500    5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028502    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028505    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028508    5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028510    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028513    5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028515    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028518    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028520    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028523    5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028525    5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028528    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028536    5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028538    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028541    5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028543    5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:20.030424 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028546    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028548    5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028551    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028554    5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028556    5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028559    5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.028564    5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029377    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029384    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029388    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029391    5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029394    5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029397    5196 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029399    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029402    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029404    5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029407    5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029409    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029412    5196 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029415    5196 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:20.030904 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029417    5196 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029420    5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029422    5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029425    5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029427    5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029430    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029432    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029434    5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029438    5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029440    5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029443    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029445    5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029448    5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029450    5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029452    5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029455    5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029458    5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029460    5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029463    5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:20.031425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029466    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029476    5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029479    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029482    5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029484    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029487    5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029489    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029492    5196 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029494    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029497    5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029499    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029502    5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029504    5196 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029507    5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029510    5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029512    5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029515    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029518    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029521    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029523    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:20.031897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029526    5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029528    5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029531    5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029533    5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029536    5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029538    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029541    5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029543    5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029545    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029548    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029550    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029553    5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029555    5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029558    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029566    5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029570    5196 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029572    5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029575    5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029577    5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:20.032409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029579    5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029582    5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029584    5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029587    5196 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029589    5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029592    5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029595    5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029598    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029600    5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029603    5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029606    5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029608    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029610    5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029613    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029616    5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029618    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029621    5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029624    5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029626    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029628    5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:20.032899 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029631    5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029634    5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029636    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029640    5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029644    5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029647    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029650    5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029652    5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029656    5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029658    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029661    5196 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029666    5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029669    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029672    5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.029675    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030232    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030239    5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030243    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030246    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:20.033492 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030248    5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030251    5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030254    5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030256    5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030259    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030262    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030265    5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030268    5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030270    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030273    5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030277    5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030281    5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030283    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030286    5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030289    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030291    5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030294    5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030297    5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030299    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:20.033979 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030302    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030304    5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030307    5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030310    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030313    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030315    5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030318    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030320    5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030323    5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030325    5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030328    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030331    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030333    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030336    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030338    5196 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030341    5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030343    5196 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030346    5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030348    5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:20.034499 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030351    5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030354    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030358    5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030362    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030367    5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030370    5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030372    5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030375    5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030378    5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030380    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030383    5196 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030385    5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030388    5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030391    5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030393    5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030395    5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030398    5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030401    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030403    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:20.034989 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030406    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030408    5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030411    5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030414    5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030417    5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030421    5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030424    5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030426    5196 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030429    5196 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030432    5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030435    5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030438    5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030440    5196 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030445    5196 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030447    5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030450    5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030452    5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030456    5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030459    5196 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030461    5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:20.035683 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030464    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030466    5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030469    5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030471    5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030473    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030476    5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030478    5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030481    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030483    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030486    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030489    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030492    5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030494    5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030496    5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030499    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030501    5196 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030504    5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030506    5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030510    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:20.036461 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030513    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030515    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030518    5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030521    5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030523    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.030525    5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030625    5196 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030639    5196 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030646    5196 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030651    5196 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030657    5196 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030660    5196 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030665    5196 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030669    5196 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030672    5196 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030675    5196 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030679    5196 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030683    5196 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030686    5196 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030688    5196 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030691    5196 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030694    5196 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030697    5196 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:29:20.036949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030700    5196 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030705    5196 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030708    5196 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030712    5196 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030715    5196 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030718    5196 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030722    5196 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030725    5196 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030728    5196 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030731    5196 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030734    5196 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030737    5196 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030741    5196 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030744    5196 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030747    5196 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030752    5196 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030755    5196 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030758    5196 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030761    5196 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030766    5196 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030768    5196 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030773    5196 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030776    5196 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030779    5196 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:29:20.037563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030782    5196 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030785    5196 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030789    5196 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030792    5196 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030795    5196 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030798    5196 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030801    5196 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030804    5196 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030807    5196 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030810    5196 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030813    5196 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030816    5196 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030819    5196 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030824    5196 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030827    5196 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030830    5196 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030832    5196 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030836    5196 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030839    5196 flags.go:64] FLAG: --help="false"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030842    5196 flags.go:64] FLAG: --hostname-override="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030846    5196 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030849    5196 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030852    5196 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:29:20.038187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030857    5196 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030861    5196 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030863    5196 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030866    5196 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030870    5196 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030873    5196 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030876    5196 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030879    5196 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030882    5196 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030885    5196 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030887    5196 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030891    5196 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030893    5196 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030896    5196 flags.go:64] FLAG: --lock-file=""
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030899    5196 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030902    5196 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030905    5196 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030910    5196 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030913    5196 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030916    5196 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030920    5196 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030923    5196 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030926    5196 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030929    5196 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030932    5196 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:29:20.038801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030937    5196 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030940    5196 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030944    5196 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030946    5196 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030949    5196 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030952    5196 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030955    5196 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030960    5196 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030963    5196 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030966    5196 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030973    5196 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030977    5196 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030980    5196 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030983    5196 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030986    5196 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030989    5196 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030992    5196 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030995    5196 flags.go:64] FLAG: --port="10250"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.030998    5196 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031001    5196 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-05a97ae4f12b6e175"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031004    5196 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031008    5196 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031011    5196 flags.go:64] FLAG: --register-node="true"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031013    5196 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031017    5196 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031020    5196 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:29:20.039441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031023    5196 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031026    5196 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031031    5196 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031035    5196 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031038    5196 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031041    5196 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031043    5196 flags.go:64] FLAG: --runonce="false"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031047    5196 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031050    5196 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031053    5196 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031055    5196 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031058    5196 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031061    5196 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031065    5196 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031069    5196 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031071    5196 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031074    5196 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031079    5196 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031082    5196 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031085    5196 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031105    5196 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031109    5196 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031115    5196 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031118    5196 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031120    5196 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:29:20.040196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031124    5196 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031127    5196 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031130    5196 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031133    5196 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031136    5196 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031139    5196 flags.go:64] FLAG: --v="2"
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031143    5196 flags.go:64] FLAG: --version="false"
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031148    5196 flags.go:64] FLAG: --vmodule=""
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031152    5196 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031155    5196 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031651    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031655    5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031658    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031661    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031665    5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031668    5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031671    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031673    5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031676    5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031678    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031683    5196 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:20.040882 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031685    5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031688    5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031690    5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031694    5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031697    5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031699    5196 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031702    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031704    5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031707    5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031709    5196 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031712    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031714    5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031718    5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031724    5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031726    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031729    5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031731    5196 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031734    5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031737    5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031739    5196 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:20.041437 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031741    5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031744    5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031747    5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031750    5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031752    5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031755    5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031757    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031760    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031762    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031765    5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031767    5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031772    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031774    5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031777    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031779    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031783    5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031786    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031788    5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031791    5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031794    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:20.041951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031796    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031799    5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031801    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031803    5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031806    5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031809    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031811    5196 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031814    5196 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031816    5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031818    5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031821    5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031823    5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031826    5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031828    5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031831    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031834    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031836    5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031838    5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031841    5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031843    5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:20.042560 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031846    5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031848    5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031851    5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031854    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031857    5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031859    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031862    5196 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031865    5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031868    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031870    5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031873    5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031875    5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031878    5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031880    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031883    5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031885    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031888    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031890    5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031893    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031895    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:20.043259 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031898    5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031900    5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031903    5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031907    5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031911    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031914    5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031916    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031919    5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031922    5196 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031925    5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031928    5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031930    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031933    5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031935    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031938    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:20.043773 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.031964    5196 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.031997    5196 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032002    5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032005    5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032008    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032011    5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032014    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032016    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032019    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032021    5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032023    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032026    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032029    5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032031    5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032034    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032036    5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032039    5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032041    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032043    5196 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:20.044161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032046    5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032049    5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032051    5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032054    5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032056    5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032058    5196 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032060    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032063    5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032066    5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032068    5196 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032071    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032073    5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032077    5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032080    5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032083    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032086    5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032105    5196 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032109    5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032114    5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032117    5196 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:20.044641 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032119    5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032122    5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032124    5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032127    5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032130    5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032132    5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032134    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032137    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032140    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032142    5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032145    5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032147    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032150    5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032152    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032155    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032157    5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032159    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032162    5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032164    5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032166    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:20.045495 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032170    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032172    5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032175    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032177    5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032179    5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032182    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032184    5196 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032187    5196 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032189    5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032191    5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032194    5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032196    5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032198    5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032201    5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032203    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032206    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032208    5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032210    5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032213    5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032215    5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:20.045997 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032218    5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032220    5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032223    5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032225    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032228    5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032230    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032232    5196 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032235    5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032238    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032240    5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032242    5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032245    5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032247    5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032250    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032253    5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032255    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032257    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032260    5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032262    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032264    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:20.046670 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032267    5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032269    5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032272    5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032276    5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032280    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032283    5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032285    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.032288    5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.037421    5196 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.037439    5196 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.037456    5196 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.037462    5196 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037524    5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037529    5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037532    5196 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037535    5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037538    5196 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037542    5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037547    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:20.047212 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037550    5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037553    5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037555    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037558    5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037560    5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037563    5196 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037566    5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037569    5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037571    5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037574    5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037577    5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037580    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037583    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037585    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037588    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037590    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037593    5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037596    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037599    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:20.047714 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037601    5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037605    5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037607    5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037610    5196 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037612    5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037615    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037617    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037620    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037623    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037625    5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037628    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037630    5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037633    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037635    5196 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037638    5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037640    5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037643    5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037646    5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037648    5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037651    5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:20.048225 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037654    5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037656    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037659    5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037661    5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037664    5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037666    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037669    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037672    5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037674    5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037677    5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037680    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037683    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037686    5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037689    5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037691    5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037694    5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037696    5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037699    5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037701    5196 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037704    5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:20.048929 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037706    5196 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037709    5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037711    5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037714    5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037716    5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037719    5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037721    5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037723    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037726    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037728    5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037731    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037733    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037736    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037739    5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037741    5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037744    5196 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037746    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037749    5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037751    5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:20.049546 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037754    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037758    5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037763    5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037766    5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037769    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037772    5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037775    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037777    5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037780    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037782    5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037784    5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037787    5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037789    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037792    5196 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037794    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037797    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037799    5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037802    5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037804    5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:20.050165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037807    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037810    5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.037840    5196 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037982    5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037986    5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037989    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037991    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037994    5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.037997    5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038000    5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038002    5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038006    5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038010    5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038013    5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038015    5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038018    5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:20.050704 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038020    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038023    5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038026    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038028    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038030    5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038033    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038035    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038038    5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038040    5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038043    5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038045    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038048    5196 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038054    5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038057    5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038060    5196 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038063    5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038065    5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038068    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038070    5196 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:20.051382 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038073    5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038075    5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038078    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038080    5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038083    5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038085    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038105    5196 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038108    5196 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038111    5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038114    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038117    5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038119    5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038122    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038124    5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038127    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038129    5196 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038132    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038135    5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038137    5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038140    5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:20.052071 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038143    5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038145    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038148    5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038151    5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038153    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038157    5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038160    5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038162    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038165    5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038167    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038170    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038173    5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038176    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038179    5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038181    5196 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038184    5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038186    5196 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038189    5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038191    5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038195    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:20.052594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038198    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038201    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038204    5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038207    5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038209    5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038211    5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038214    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038217    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038219    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038222    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038224    5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038226    5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038229    5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038231    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038234    5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038236    5196 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038239    5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038246    5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038248    5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:20.053192 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038251    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038253    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038256    5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038258    5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038261    5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038265    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038267    5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038269    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038272    5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038274    5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038277    5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038280    5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038282    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038285    5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038288    5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:20.053675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.038313    5196 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038348    5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038352    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038355    5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038358    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038360    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038363    5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038365    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038368    5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038370    5196 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038373    5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038375    5196 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038378    5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038380    5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038383    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038385    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038389    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038392    5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038395    5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038397    5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:20.054046 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038400    5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038402    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038405    5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038407    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038410    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038413    5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038415    5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038418    5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038420    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038422    5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038425    5196 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038428    5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038431    5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038433    5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038436    5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038438    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038440    5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038443    5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038446    5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:20.054547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038449    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038451    5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038454    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038456    5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038459    5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038461    5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038464    5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038466    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038469    5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038473    5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038476    5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038478    5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038481    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038483    5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038486    5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038489    5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038491    5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038494    5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038498    5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:20.055015 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038501    5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038504    5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038507    5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038509    5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038512    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038514    5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038518    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038520    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038523    5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038525    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038528    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038531    5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038533    5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038536    5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038538    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038541    5196 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038545    5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038549    5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038551    5196 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:20.055528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038554    5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038556    5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038559    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038563    5196 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038565    5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038568    5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038570    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038573    5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038576    5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038578    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038581    5196 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038583    5196 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038586    5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038588    5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038591    5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038594    5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038596    5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038599    5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038601    5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038604    5196 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:20.055983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038607    5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038610    5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038612    5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038615    5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038617    5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038620    5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038623    5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038625    5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038628    5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:20.038631    5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.038692    5196 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.039672    5196 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.039731    5196 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.039914    5196 server.go:1014] "Starting client certificate rotation"
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.040062    5196 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.040153    5196 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:03 +0000 UTC" deadline="2028-05-03 20:18:42.478443882 +0000 UTC"
Nov 03 20:29:20.056545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.040203    5196 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13127h49m22.438244215s"
Nov 03 20:29:20.057173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.040460    5196 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:29:20.057173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.041636    5196 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:29:20.057173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.042823    5196 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:29:20.057173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.044551    5196 log.go:25] "Validated CRI v1 image API"
Nov 03 20:29:20.057173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.045385    5196 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:29:20.057173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.047304    5196 fs.go:135] Filesystem UUIDs: map[32f7ec49-494c-4613-87f4-9633b0e626a8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d381774a-e4ab-4d38-b2d2-cf7422b6ee02:/dev/nvme0n1p3]
Nov 03 20:29:20.057378 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.047320    5196 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0038063b6dccb41ed059b4906cdaa9066b35b3a7b48e4550e165cbb8d954662e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0038063b6dccb41ed059b4906cdaa9066b35b3a7b48e4550e165cbb8d954662e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04eba560ee1d9cc80f006d14e5af55b65c817d405a813190c4f6f432e63186d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04eba560ee1d9cc80f006d14e5af55b65c817d405a813190c4f6f432e63186d4/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fc02f488b7cfdccfb2ecf6ddbdaccd8f061781dfc70e205d77d08955c0ea774/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fc02f488b7cfdccfb2ecf6ddbdaccd8f061781dfc70e205d77d08955c0ea774/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25ff54de6d84ef9481b656197a9c26c6809faf59c769cfb8ccf74a7ea156ed4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25ff54de6d84ef9481b656197a9c26c6809faf59c769cfb8ccf74a7ea156ed4d/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/261cff70c841b87369b2156d17642a305c347764a822b98bb5648467d24ab81a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/261cff70c841b87369b2156d17642a305c347764a822b98bb5648467d24ab81a/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33e2c1bfb20b02194555408becd8026b73a838f919064c807c2790f11efa3df0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33e2c1bfb20b02194555408becd8026b73a838f919064c807c2790f11efa3df0/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37c190292dca8dd29400317822932d7169a10879152a603eab7442e65203d8c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37c190292dca8dd29400317822932d7169a10879152a603eab7442e65203d8c3/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d09e80183bf9d2b72eecb5a3e06b8aff84d47ad81755d84c86e4798c5e83d0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d09e80183bf9d2b72eecb5a3e06b8aff84d47ad81755d84c86e4798c5e83d0e/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f367a1db172495b6833df570530ad42ec86a1a15b9e829f793190dc2a612074/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f367a1db172495b6833df570530ad42ec86a1a15b9e829f793190dc2a612074/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4189997ef87adf82a22a2bf89dcfae62b830dff5cf02b1c47ff8d0dd37368e3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4189997ef87adf82a22a2bf89dcfae62b830dff5cf02b1c47ff8d0dd37368e3d/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42807ab1243baaaf868bbd9e9c8f7d44ed57289e344056e2aa66632feb19e2ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42807ab1243baaaf868bbd9e9c8f7d44ed57289e344056e2aa66632feb19e2ef/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52c7947ad1b050cd7270bcf7dd5c79119339283eb750f1b0bb567ea8f4bcb780/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52c7947ad1b050cd7270bcf7dd5c79119339283eb750f1b0bb567ea8f4bcb780/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d34b35a471354374ecdafb1189d9ef8ffd3fe4417465c91072d4b805f10eb4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d34b35a471354374ecdafb1189d9ef8ffd3fe4417465c91072d4b805f10eb4e/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f141359905d68ece3a56a2690311611483057b67128884a1304e61f0038bb73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f141359905d68ece3a56a2690311611483057b67128884a1304e61f0038bb73/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87a23b206956969cdaf1fb7169aed87ea8b90f8dbb513cf357704e9393654eda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87a23b206956969cdaf1fb7169aed87ea8b90f8dbb513cf357704e9393654eda/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9609b0c919b7fb3ea6f60ceb0da38f0315b03ca8f64f8b8227babc52c8b8df60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9609b0c919b7fb3ea6f60ceb0da38f0315b03ca8f64f8b8227babc52c8b8df60/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ac08017bcf4ab7a41e54278af39a59868dc4eccf639011e5b8f457ac05b2b39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ac08017bcf4ab7a41e54278af39a59868dc4eccf639011e5b8f457ac05b2b39/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a138a24953b025e246d185cb92a3f4476bbba2e1df99bcd83ada96f5bf60d382/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a138a24953b025e246d185cb92a3f4476bbba2e1df99bcd83ada96f5bf60d382/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a70653862c65783018189dfc9eaeac628cd2d7eaf59223862514c793002c953e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a70653862c65783018189dfc9eaeac628cd2d7eaf59223862514c793002c953e/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5af1f37546b34aaf445847294f005402ecab75b88ba4aa4dff65991ab22c41b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5af1f37546b34aaf445847294f005402ecab75b88ba4aa4dff65991ab22c41b/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf564e89cef3b9789adbccfadee26383b08448b87ddab3cef870381a6d2676b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf564e89cef3b9789adbccfadee26383b08448b87ddab3cef870381a6d2676b4/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0174d5a15736fa727b3a1ccb3f54e910ae13fca12c0c1abdf516665051a79b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0174d5a15736fa727b3a1ccb3f54e910ae13fca12c0c1abdf516665051a79b2/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/effde35f08fb9873c7d2e4336acffd86d749d9008847ef31adca6c8cc5882f9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/effde35f08fb9873c7d2e4336acffd86d749d9008847ef31adca6c8cc5882f9b/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1113fe81-67a7-4c02-ae59-d9fd15e22bd0/volumes/kubernetes.io~projected/kube-api-access-nmqtz:{mountpoint:/var/lib/kubelet/pods/1113fe81-67a7-4c02-ae59-d9fd15e22bd0/volumes/kubernetes.io~projected/kube-api-access-nmqtz major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/151b0e6d-c5a6-469c-8d77-f7507ec6796a/volumes/kubernetes.io~projected/kube-api-access-nlcpg:{mountpoint:/var/lib/kubelet/pods/151b0e6d-c5a6-469c-8d77-f7507ec6796a/volumes/kubernetes.io~projected/kube-api-access-nlcpg major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/151b0e6d-c5a6-469c-8d77-f7507ec6796a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/151b0e6d-c5a6-469c-8d77-f7507ec6796a/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/170428c7-02e4-40f5-a88c-1634d8b58b5a/volumes/kubernetes.io~projected/kube-api-access-8dpww:{mountpoint:/var/lib/kubelet/pods/170428c7-02e4-40f5-a88c-1634d8b58b5a/volumes/kubernetes.io~projected/kube-api-access-8dpww major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/170428c7-02e4-40f5-a88c-1634d8b58b5a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/170428c7-02e4-40f5-a88c-1634d8b58b5a/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17826854-67bc-4852-97c1-83ebe2c648f7/volumes/kubernetes.io~projected/kube-api-access-wmt7s:{mountpoint:/var/lib/kubelet/pods/17826854-67bc-4852-97c1-83ebe2c648f7/volumes/kubernetes.io~projected/kube-api-access-wmt7s major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d9f21de-9ec8-48b7-aa66-5945a79d95e4/volumes/kubernetes.io~projected/kube-api-access-rpgb4:{mountpoint:/var/lib/kubelet/pods/1d9f21de-9ec8-48b7-aa66-5945a79d95e4/volumes/kubernetes.io~projected/kube-api-access-rpgb4 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/428ef791-c562-4786-b095-7c3b1113bfc8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/428ef791-c562-4786-b095-7c3b1113bfc8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46ca8995-d9ba-42fb-94b6-d7cf18696cce/volumes/kubernetes.io~projected/kube-api-access-46nx5:{mountpoint:/var/lib/kubelet/pods/46ca8995-d9ba-42fb-94b6-d7cf18696cce/volumes/kubernetes.io~projected/kube-api-access-46nx5 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46ca8995-d9ba-42fb-94b6-d7cf18696cce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/46ca8995-d9ba-42fb-94b6-d7cf18696cce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/492e27da-7982-4035-945f-066588cfdb79/volumes/kubernetes.io~projected/kube-api-access-mmp24:{mountpoint:/var/lib/kubelet/pods/492e27da-7982-4035-945f-066588cfdb79/volumes/kubernetes.io~projected/kube-api-access-mmp24 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~projected/kube-api-access-mmtt5:{mountpoint:/var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~projected/kube-api-access-mmtt5 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53ee9fcc-846d-4ca8-835f-58d77e3f1a63/volumes/kubernetes.io~projected/kube-api-access-nvpl4:{mountpoint:/var/lib/kubelet/pods/53ee9fcc-846d-4ca8-835f-58d77e3f1a63/volumes/kubernetes.io~projected/kube-api-access-nvpl4 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/545aca01-0ea8-4df2-b3bc-fa60f6f1c725/volumes/kubernetes.io~projected/kube-api-access-5ssd9:{mountpoint:/var/lib/kubelet/pods/545aca01-0ea8-4df2-b3bc-fa60f6f1c725/volumes/kubernetes.io~projected/kube-api-access-5ssd9 major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68c418d8-0eb1-44a4-9c0f-59d3d7d724af/volumes/kubernetes.io~projected/kube-api-access-xt5pj:{mountpoint:/var/lib/kubelet/pods/68c418d8-0eb1-44a4-9c0f-59d3d7d724af/volumes/kubernetes.io~projected/kube-api-access-xt5pj major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68f3b7ee-a558-4c4d-8625-11f72d13b257/volumes/kubernetes.io~projected/kube-api-access-qkz4p:{mountpoint:/var/lib/kubelet/pods/68f3b7ee-a558-4c4d-8625-11f72d13b257/volumes/kubernetes.io~projected/kube-api-access-qkz4p major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a/volumes/kubernetes.io~projected/kube-api-access-7wvg9:{mountpoint:/var/lib/kubelet/pods/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a/volumes/kubernetes.io~projected/kube-api-access-7wvg9 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volumes/kubernetes.io~projected/kube-api-access-m2b2l:{mountpoint:/var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volumes/kubernetes.io~projected/kube-api-access-m2b2l major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d8dc680-3f86-4ec8-949a-18cb41c680d1/volumes/kubernetes.io~projected/kube-api-access-2xr94:{mountpoint:/var/lib/kubelet/pods/9d8dc680-3f86-4ec8-949a-18cb41c680d1/volumes/kubernetes.io~projected/kube-api-access-2xr94 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f9c06cb-46f7-440b-b395-a4c4e9a597cd/volumes/kubernetes.io~projected/kube-api-access-mqk6q:{mountpoint:/var/lib/kubelet/pods/9f9c06cb-46f7-440b-b395-a4c4e9a597cd/volumes/kubernetes.io~projected/kube-api-access-mqk6q major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f9c06cb-46f7-440b-b395-a4c4e9a597cd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9f9c06cb-46f7-440b-b395-a4c4e9a597cd/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3b4704b-2627-4530-8100-838f2d8b7d31/volumes/kubernetes.io~projected/kube-api-access-fdcfn:{mountpoint:/var/lib/kubelet/pods/a3b4704b-2627-4530-8100-838f2d8b7d31/volumes/kubernetes.io~projected/kube-api-access-fdcfn major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5/volumes/kubernetes.io~projected/kube-api-access-c8xkj:{mountpoint:/var/lib/kubelet/pods/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5/volumes/kubernetes.io~projected/kube-api-access-c8xkj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a533118b-20c1-478a-b7e7-42dd7b4f200c/volumes/kubernetes.io~projected/kube-api-access-szqjh:{mountpoint:/var/lib/kubelet/pods/a533118b-20c1-478a-b7e7-42dd7b4f200c/volumes/kubernetes.io~projected/kube-api-access-szqjh major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abf3688b-7862-4d8c-8e8f-03a791523655/volumes/kubernetes.io~projected/kube-api-access-dcz4p:{mountpoint:/var/lib/kubelet/pods/abf3688b-7862-4d8c-8e8f-03a791523655/volumes/kubernetes.io~projected/kube-api-access-dcz4p major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfdf39cf-ae26-400e-b929-80512e786f5a/volumes/kubernetes.io~projected/kube-api-access-tpqfx:{mountpoint:/var/lib/kubelet/pods/bfdf39cf-ae26-400e-b929-80512e786f5a/volumes/kubernetes.io~projected/kube-api-access-tpqfx major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfdf39cf-ae26-400e-b929-80512e786f5a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bfdf39cf-ae26-400e-b929-80512e786f5a/volumes/kubernetes.io~secret/cert major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3418cdf-4a4a-426f-bdcc-140c84c48a5a/volumes/kubernetes.io~projected/kube-api-access-sl9zc:{mountpoint:/var/lib/kubelet/pods/c3418cdf-4a4a-426f-bdcc-140c84c48a5a/volumes/kubernetes.io~projected/kube-api-access-sl9zc major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6/volumes/kubernetes.io~projected/kube-api-access-xp8gd:{mountpoint:/var/lib/kubelet/pods/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6/volumes/kubernetes.io~projected/kube-api-access-xp8gd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b325b4-c286-465f-83c9-623e0d441d17/volumes/kubernetes.io~projected/kube-api-access-fwc28:{mountpoint:/var/lib/kubelet/pods/e8b325b4-c286-465f-83c9-623e0d441d17/volumes/kubernetes.io~projected/kube-api-access-fwc28 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b325b4-c286-465f-83c9-623e0d441d17/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e8b325b4-c286-465f-83c9-623e0d441d17/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~projected/kube-api-access-s8sqv:{mountpoint:/var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~projected/kube-api-access-s8sqv major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~secret/default-certificate major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/79bd878332e2532575415410850e9033a185a49749d1ff1bbdaf2bed52a31b44/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/746fd277eb5002b3b05be77a751d0e5313fb1e1c2f7e39aa15963f12aee02b7a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8368124a62e40e6a18e91dab488bad93b2b1a3324dd0e9164a15b86c25dc18c6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/61dc81db0eaaaa54b4efd1f16be08b07de5e2f6b908f606405c935d82f9eaa84/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3fa9b36c8e78a50ca36e9a2eb5fcecb951c31780ee54ac327fd6793faf64f6d9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2cc74676aed8e01f665e5f1d74bc30ccb3df2dcd9f37450f5710e3ca1755e80a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bf677868af0f0aa643df8cf1fcd56005270bbe0805aaea3bc94917915dff382d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1b41a24cf0eae86726f3345ba2db40f729311bae6fbf5b56a82041e7e3d05d56/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b8b3db2304c899973364e164dba3a4c0c0123df4cc628a7935bd7f0cfe65fff2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/bf4d510913e0405bf5b1fecad7863bd12dd94f8c27f0320db276bdfd18b6ba6c/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/d37d25087de419951780a19c5707c84adee6d66d22e51952b781f9377fb2acc9/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/fd0a6f1c7b45afd5d18d381ac496a9d31683dee4fa7d98bebfd3eec950f4b0da/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/63a73b4f57229553e1c3ef8c6f31356bf8c5430cdbf6885493e05be0a49a0c94/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5223c4402c0d1cdcae2bee7427e04d713152009dbd2a61d87eafed08bb511b6b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e361f079c8b91fd044c16f10e1719e8526fc26b24b2c16fa924a996653656ea1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/adc05adb99ed2ab42fcc84f6635154cabbd2a19f40431641c5a30558043d0a24/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b1e73969eb73e2a52fbaf062b4f1ca1f3669d1f1916c1ddbd854c609ce8189c4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9cc90b087d1e29f9251017c11072ee946492fca35aa8c5da9eeacba49a13769a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/47d9b4344a80fdc3545c9665c2d7f5ea2fa933ba2bd727598317885e98443f2a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d6c21fcdc0480f973f682453375566e2db370695f14790ea167cda9f7a6a9266/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d06c856c04aceac0d8019b476be27208884098dd31403b2ff0f5444fd297f02c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/46677883e909a677a5887142a8203b3d64469be1db1faab5b2699863cd07d563/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/c1f1238d45dec339a3ef3d77438fea8a836ad03281cbabf7b9a90853cf3ac4f7/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/f1c3bc1ffceab5e9e6feb122b943903e12d452acd24744afd47cb658bb95c69a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/a8610e75ec9c2479ee12680724fde8f5aa5b5308004aa8e178cab2edf288af02/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/64ec132aa598a47239230efce2f3e2727a8619766377a244ddb891b9f7d86303/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f8f967c98195ef5661858ae33b33848a2cf7c7022ac0b9c928c2481bfdd0a3ad/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/9b025f24a397533acf45fb0ac84fd26e2e4aaaa56dea2fc52432de47527b4efb/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9e873291faf3bcc7ef823329d1b6c4cdc1c4480ab438f730d332240bb5a57525/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/121edf8f25cfe9c4a144d5bd6d5c4a49ecc29e62035dc66ea620bf7fbd7a7935/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/bea0eca71e26c03fb3a4815eb77842f5798622d34647a7de83819717b1193535/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/b8d60590c56f3796a70fba79bc398d0de250395f536860a32466c4730b38066d/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/5776dd0db748bf8945c6df8f473e2e975880f7df08386aa50b042be805731b2a/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/8254e32c9cf62fc3d680e94dbd2607cdb10b1342477b010df30e6fbc6a217ea9/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/fe7e0b7b6f4ae188536cce71eb2281fead0947721065463dcf315098639a1771/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/974f2993a228e69f0bf33fc763a36ec343f003a71dbfbb554a16ee534327b75a/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/73700b395415b1f38779c110a07b0074e3c79f2506ef8e75b366c5268e78b602/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/43f1108203fc44b563c9c4b28db7f128e4b25d6876d0701e32c6dd13af10dc68/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/b734d7a8e7c7cd0383fa6e08ddb9fa7f385960c4fa654ca21a5f4e7438061619/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/97bb2ed98ac53d50c4ba82d229f9e5159938980faa5363b30fa2c494dfae0270/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/37c4051f4b34e057b23aabbe7506594c292a06517b99297db4320e1bfbcaf4e6/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/fb3f733c8e669791ebdfd5786f3dbe2eac0f1fa1528ce7eb3438ef767bcf5889/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/d06d43dc3743550283dc9de35a5b79ba129e22ff3665eff78b58ff09e4aa05ce/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a11da402257c5e14579c0e65b86485e790ac5ff6a5eb80b5e38f3764daa35888/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/2ccd563ce7da2ccdb104c02cb252697a2735cca84d978a96f104bf8ecbbc68c3/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/2f021d3a41d5bd6e2c12a54b5ad465f063fd1cab68b23b9e5906950d6ba3b24c/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/b975830fbc337932e724eb334acf416a1aa716d8a8307f78afb3e9a000e1293d/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/404dbd48c16c1e435b4cd87bbf028318395a95cfdbd65d67a3f73c05ae53b766/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/fd7d9c72e2cb1ed7f6b9c0c96d5c7e0b3adf5690c6f39c43ef46c1b96785e48f/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/40c2e69fce45d3d902c8f46642b54b41355f05d96a746070c4252905ab32615c/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/908514f60fd33a58f1761e5fc77d1a1b436575e26852faf440f065720a473207/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/174ab43ad1a09825c63d2b6b9e7c9016263cd01b7ed4bdafbc00fe81c23aa4d1/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/c827d721574dbfc59e3490f6400172481e344b14246a8571c3d6b724c4bee492/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/15e1963a3f2b0ce5487e106ab5bd0121bf284661644e2a9eadec6a1036b503ad/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d436ee068a2148c799f453ebf1912710304e59496b7ef62e3294037d2420b5b9/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/515fd2fcf01124dc0e3e3448d3d62b851883c1c406309e9c0edb184b3f226476/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/63851f7a98c507a0c75fd994a5385826baa914e4e17d266295d927fba36f6d1b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7ec137a1f884814b5890ee6a3ecdd8ce47307f4981c1d24f19b17a5201349cb9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.055569    5196 manager.go:217] Machine: {Timestamp:2026-11-03 20:29:20.055083245 +0000 UTC m=+0.059041960 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100269 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282563511182b3d5c8715dd68c75ef SystemUUID:ec282563-5111-82b3-d5c8-715dd68c75ef BootID:844b484a-c414-41a1-b6f3-9bf925f732b7 Filesystems:[{Device:/run/containers/storage/overlay-containers/04eba560ee1d9cc80f006d14e5af55b65c817d405a813190c4f6f432e63186d4/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1113fe81-67a7-4c02-ae59-d9fd15e22bd0/volumes/kubernetes.io~projected/kube-api-access-nmqtz DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4189997ef87adf82a22a2bf89dcfae62b830dff5cf02b1c47ff8d0dd37368e3d/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/545aca01-0ea8-4df2-b3bc-fa60f6f1c725/volumes/kubernetes.io~projected/kube-api-access-5ssd9 DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42807ab1243baaaf868bbd9e9c8f7d44ed57289e344056e2aa66632feb19e2ef/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b325b4-c286-465f-83c9-623e0d441d17/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/68c418d8-0eb1-44a4-9c0f-59d3d7d724af/volumes/kubernetes.io~projected/kube-api-access-xt5pj DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a70653862c65783018189dfc9eaeac628cd2d7eaf59223862514c793002c953e/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87a23b206956969cdaf1fb7169aed87ea8b90f8dbb513cf357704e9393654eda/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46ca8995-d9ba-42fb-94b6-d7cf18696cce/volumes/kubernetes.io~projected/kube-api-access-46nx5 DeviceMajor:0 DeviceMinor:317 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0174d5a15736fa727b3a1ccb3f54e910ae13fca12c0c1abdf516665051a79b2/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/151b0e6d-c5a6-469c-8d77-f7507ec6796a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/170428c7-02e4-40f5-a88c-1634d8b58b5a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/492e27da-7982-4035-945f-066588cfdb79/volumes/kubernetes.io~projected/kube-api-access-mmp24 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f9c06cb-46f7-440b-b395-a4c4e9a597cd/volumes/kubernetes.io~projected/kube-api-access-mqk6q DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1d9f21de-9ec8-48b7-aa66-5945a79d95e4/volumes/kubernetes.io~projected/kube-api-access-rpgb4 DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/428ef791-c562-4786-b095-7c3b1113bfc8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37c190292dca8dd29400317822932d7169a10879152a603eab7442e65203d8c3/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33e2c1bfb20b02194555408becd8026b73a838f919064c807c2790f11efa3df0/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf564e89cef3b9789adbccfadee26383b08448b87ddab3cef870381a6d2676b4/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/151b0e6d-c5a6-469c-8d77-f7507ec6796a/volumes/kubernetes.io~projected/kube-api-access-nlcpg DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a138a24953b025e246d185cb92a3f4476bbba2e1df99bcd83ada96f5bf60d382/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bfdf39cf-ae26-400e-b929-80512e786f5a/volumes/kubernetes.io~projected/kube-api-access-tpqfx DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68f3b7ee-a558-4c4d-8625-11f72d13b257/volumes/kubernetes.io~projected/kube-api-access-qkz4p DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a533118b-20c1-478a-b7e7-42dd7b4f200c/volumes/kubernetes.io~projected/kube-api-access-szqjh DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5/volumes/kubernetes.io~projected/kube-api-access-c8xkj DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0038063b6dccb41ed059b4906cdaa9066b35b3a7b48e4550e165cbb8d954662e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d8dc680-3f86-4ec8-949a-18cb41c680d1/volumes/kubernetes.io~projected/kube-api-access-2xr94 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f141359905d68ece3a56a2690311611483057b67128884a1304e61f0038bb73/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f9c06cb-46f7-440b-b395-a4c4e9a597cd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3b4704b-2627-4530-8100-838f2d8b7d31/volumes/kubernetes.io~projected/kube-api-access-fdcfn DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a/volumes/kubernetes.io~projected/kube-api-access-7wvg9 DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d09e80183bf9d2b72eecb5a3e06b8aff84d47ad81755d84c86e4798c5e83d0e/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f367a1db172495b6833df570530ad42ec86a1a15b9e829f793190dc2a612074/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6/volumes/kubernetes.io~projected/kube-api-access-xp8gd DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee248b83-7c43-46fe-809c-69b61ca6254a/volumes/kubernetes.io~projected/kube-api-access-s8sqv DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/261cff70c841b87369b2156d17642a305c347764a822b98bb5648467d24ab81a/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ac08017bcf4ab7a41e54278af39a59868dc4eccf639011e5b8f457ac05b2b39/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46ca8995-d9ba-42fb-94b6-d7cf18696cce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfdf39cf-ae26-400e-b929-80512e786f5a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/effde35f08fb9873c7d2e4336acffd86d749d9008847ef31adca6c8cc5882f9b/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9609b0c919b7fb3ea6f60ceb0da38f0315b03ca8f64f8b8227babc52c8b8df60/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17826854-67bc-4852-97c1-83ebe2c648f7/volumes/kubernetes.io~projected/kube-api-access-wmt7s DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5af1f37546b34aaf445847294f005402ecab75b88ba4aa4dff65991ab22c41b/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/170428c7-02e4-40f5-a88c-1634d8b58b5a/volumes/kubernetes.io~projected/kube-api-access-8dpww DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/53ee9fcc-846d-4ca8-835f-58d77e3f1a63/volumes/kubernetes.io~projected/kube-api-access-nvpl4 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/volumes/kubernetes.io~projected/kube-api-access-m2b2l DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52c7947ad1b050cd7270bcf7dd5c79119339283eb750f1b0bb567ea8f4bcb780/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abf3688b-7862-4d8c-8e8f-03a791523655/volumes/kubernetes.io~projected/kube-api-access-dcz4p DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d34b35a471354374ecdafb1189d9ef8ffd3fe4417465c91072d4b805f10eb4e/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/4b434899-5102-48c6-a2cf-c37ccd545f8a/volumes/kubernetes.io~projected/kube-api-access-mmtt5 DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25ff54de6d84ef9481b656197a9c26c6809faf59c769cfb8ccf74a7ea156ed4d/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c3418cdf-4a4a-426f-bdcc-140c84c48a5a/volumes/kubernetes.io~projected/kube-api-access-sl9zc DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fc02f488b7cfdccfb2ecf6ddbdaccd8f061781dfc70e205d77d08955c0ea774/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b325b4-c286-465f-83c9-623e0d441d17/volumes/kubernetes.io~projected/kube-api-access-fwc28 DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04eba560ee1d9cc MacAddress:de:6b:c9:09:04:aa Speed:10000 Mtu:8901} {Name:261cff70c841b87 MacAddress:4e:8c:33:89:17:c1 Speed:10000 Mtu:8901} {Name:37c190292dca8dd MacAddress:16:dc:30:31:04:3b Speed:10000 Mtu:8901} {Name:3d09e80183bf9d2 MacAddress:86:99:65:c9:b6:39 Speed:10000 Mtu:8901} {Name:4189997ef87adf8 MacAddress:8e:4a:6c:5a:1e:3b Speed:10000 Mtu:8901} {Name:52c7947ad1b050c MacAddress:32:fb:ba:69:10:50 Speed:10000 Mtu:8901} {Name:6d34b35a4713543 MacAddress:ee:78:9b:b9:4a:38 Speed:10000 Mtu:8901} {Name:87a23b206956969 MacAddress:b6:68:50:34:ec:33 Speed:10000 Mtu:8901} {Name:9609b0c919b7fb3 MacAddress:36:a8:73:01:6b:19 Speed:10000 Mtu:8901} {Name:a138a24953b025e MacAddress:ee:2c:1a:c5:1a:04 Speed:10000 Mtu:8901} {Name:a70653862c65783 MacAddress:4a:70:c3:8a:65:ea Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d3:b4:56:d3 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:29:41:2a:df:09 Speed:0 Mtu:8901} {Name:c5af1f37546b34a MacAddress:e2:e5:a1:1c:1a:af Speed:10000 Mtu:8901} {Name:effde35f08fb987 MacAddress:72:92:bf:18:d5:3e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d3:b4:56:d3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:86:58:89:98:08:72 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:33:17:3a:62:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.055919    5196 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.055969    5196 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056274    5196 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056301    5196 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-49.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056432    5196 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056440    5196 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056451    5196 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056471    5196 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056489    5196 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056646    5196 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056764    5196 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056799    5196 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056813    5196 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056828    5196 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056841    5196 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:29:20.057563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.056853    5196 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:29:20.058268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.057581    5196 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:29:20.058268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.057591    5196 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:29:20.058268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.058153    5196 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:29:20.060402 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060388    5196 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:29:20.060443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060402    5196 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:29:20.060518 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060508    5196 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 20:29:20.060711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060700    5196 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:29:20.060742 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060730    5196 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:29:20.060832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060822    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:29:20.060861 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060843    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:29:20.060888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060870    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:29:20.060915 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060892    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:29:20.060915 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060905    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:29:20.060971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060917    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:29:20.060971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060931    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:29:20.060971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060944    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:29:20.060971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060961    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:29:20.060971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060970    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:29:20.061287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060984    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:29:20.061287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.060995    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:29:20.061287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.061024    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:29:20.061287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.061034    5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:29:20.064582 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.064569    5196 server.go:1263] "Started kubelet"
Nov 03 20:29:20.064743 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.064678    5196 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:29:20.065476 ip-10-0-128-49 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:29:20.065601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.065510    5196 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:29:20.065672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.065616    5196 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:29:20.068190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.068143    5196 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:29:20.069297 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.069275    5196 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:29:20.071935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.070604    5196 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:29:20.071935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.070697    5196 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:29:20.071935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.070747    5196 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:29:20.071935 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.070731    5196 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-49.us-west-2.compute.internal\" not found"
Nov 03 20:29:20.071935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.070814    5196 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:04 +0000 UTC" deadline="2028-07-02 02:09:13.175938547 +0000 UTC"
Nov 03 20:29:20.071935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.070852    5196 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14549h39m53.10509247s"
Nov 03 20:29:20.071935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.071020    5196 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:29:20.073453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.073429    5196 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:29:20.074055 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.074032    5196 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:29:20.074151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.074056    5196 factory.go:55] Registering systemd factory
Nov 03 20:29:20.074151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.074068    5196 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:29:20.075199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075182    5196 factory.go:153] Registering CRI-O factory
Nov 03 20:29:20.075199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075173    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151b0e6d-c5a6-469c-8d77-f7507ec6796a" volumeName="kubernetes.io/projected/151b0e6d-c5a6-469c-8d77-f7507ec6796a-kube-api-access-nlcpg" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075213    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="170428c7-02e4-40f5-a88c-1634d8b58b5a" volumeName="kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-config" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075227    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68c418d8-0eb1-44a4-9c0f-59d3d7d724af" volumeName="kubernetes.io/projected/68c418d8-0eb1-44a4-9c0f-59d3d7d724af-kube-api-access-xt5pj" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075239    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68f3b7ee-a558-4c4d-8625-11f72d13b257" volumeName="kubernetes.io/configmap/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-config" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075250    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88128ae2-7d2c-41a4-a0d1-ceb286a7d80f" volumeName="kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-env-overrides" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075262    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b325b4-c286-465f-83c9-623e0d441d17" volumeName="kubernetes.io/configmap/e8b325b4-c286-465f-83c9-623e0d441d17-config" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075202    5196 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075272    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b325b4-c286-465f-83c9-623e0d441d17" volumeName="kubernetes.io/projected/e8b325b4-c286-465f-83c9-623e0d441d17-kube-api-access-fwc28" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075284    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="170428c7-02e4-40f5-a88c-1634d8b58b5a" volumeName="kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-trusted-ca" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075296    5196 factory.go:103] Registering Raw factory
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075300    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3b4704b-2627-4530-8100-838f2d8b7d31" volumeName="kubernetes.io/projected/a3b4704b-2627-4530-8100-838f2d8b7d31-kube-api-access-fdcfn" seLinuxMountContext=""
Nov 03 20:29:20.075308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075311    5196 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075313    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf3688b-7862-4d8c-8e8f-03a791523655" volumeName="kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075325    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46ca8995-d9ba-42fb-94b6-d7cf18696cce" volumeName="kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075338    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d8dc680-3f86-4ec8-949a-18cb41c680d1" volumeName="kubernetes.io/configmap/9d8dc680-3f86-4ec8-949a-18cb41c680d1-serviceca" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075349    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88128ae2-7d2c-41a4-a0d1-ceb286a7d80f" volumeName="kubernetes.io/secret/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075363    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f9c06cb-46f7-440b-b395-a4c4e9a597cd" volumeName="kubernetes.io/projected/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-kube-api-access-mqk6q" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075400    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3b4704b-2627-4530-8100-838f2d8b7d31" volumeName="kubernetes.io/configmap/a3b4704b-2627-4530-8100-838f2d8b7d31-telemetry-config" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075411    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b325b4-c286-465f-83c9-623e0d441d17" volumeName="kubernetes.io/secret/e8b325b4-c286-465f-83c9-623e0d441d17-serving-cert" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075424    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d9f21de-9ec8-48b7-aa66-5945a79d95e4" volumeName="kubernetes.io/projected/1d9f21de-9ec8-48b7-aa66-5945a79d95e4-kube-api-access-rpgb4" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075439    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="859d83db-cfcc-459f-8ed2-8ec6a7e88b1a" volumeName="kubernetes.io/projected/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a-kube-api-access-7wvg9" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075452    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cabc49-e7ed-4c2a-8652-114a52f0f7a5" volumeName="kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075466    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cabc49-e7ed-4c2a-8652-114a52f0f7a5" volumeName="kubernetes.io/projected/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-kube-api-access-c8xkj" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075479    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a533118b-20c1-478a-b7e7-42dd7b4f200c" volumeName="kubernetes.io/projected/a533118b-20c1-478a-b7e7-42dd7b4f200c-kube-api-access-szqjh" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075493    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf3688b-7862-4d8c-8e8f-03a791523655" volumeName="kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075505    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151b0e6d-c5a6-469c-8d77-f7507ec6796a" volumeName="kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-snapshots" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075518    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="428ef791-c562-4786-b095-7c3b1113bfc8" volumeName="kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075533    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b434899-5102-48c6-a2cf-c37ccd545f8a" volumeName="kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-tmp" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075545    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53ee9fcc-846d-4ca8-835f-58d77e3f1a63" volumeName="kubernetes.io/projected/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-kube-api-access-nvpl4" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075559    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3" volumeName="kubernetes.io/secret/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-agent-certs" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075577    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f9c06cb-46f7-440b-b395-a4c4e9a597cd" volumeName="kubernetes.io/secret/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-serving-cert" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075599    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf3688b-7862-4d8c-8e8f-03a791523655" volumeName="kubernetes.io/projected/abf3688b-7862-4d8c-8e8f-03a791523655-kube-api-access-dcz4p" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075617    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3" volumeName="kubernetes.io/configmap/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-konnectivity-ca" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075633    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="545aca01-0ea8-4df2-b3bc-fa60f6f1c725" volumeName="kubernetes.io/configmap/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-config-volume" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075647    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68f3b7ee-a558-4c4d-8625-11f72d13b257" volumeName="kubernetes.io/projected/68f3b7ee-a558-4c4d-8625-11f72d13b257-kube-api-access-qkz4p" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075651    5196 manager.go:319] Starting recovery of all containers
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075659    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6" volumeName="kubernetes.io/projected/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-kube-api-access-xp8gd" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075700    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151b0e6d-c5a6-469c-8d77-f7507ec6796a" volumeName="kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-tmp" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075718    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17826854-67bc-4852-97c1-83ebe2c648f7" volumeName="kubernetes.io/projected/17826854-67bc-4852-97c1-83ebe2c648f7-kube-api-access-wmt7s" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075735    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46ca8995-d9ba-42fb-94b6-d7cf18696cce" volumeName="kubernetes.io/projected/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-api-access-46nx5" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075763    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b434899-5102-48c6-a2cf-c37ccd545f8a" volumeName="kubernetes.io/projected/4b434899-5102-48c6-a2cf-c37ccd545f8a-kube-api-access-mmtt5" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075778    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="545aca01-0ea8-4df2-b3bc-fa60f6f1c725" volumeName="kubernetes.io/empty-dir/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-tmp-dir" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075791    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfdf39cf-ae26-400e-b929-80512e786f5a" volumeName="kubernetes.io/secret/bfdf39cf-ae26-400e-b929-80512e786f5a-cert" seLinuxMountContext=""
Nov 03 20:29:20.075784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075803    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee248b83-7c43-46fe-809c-69b61ca6254a" volumeName="kubernetes.io/projected/ee248b83-7c43-46fe-809c-69b61ca6254a-kube-api-access-s8sqv" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075816    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151b0e6d-c5a6-469c-8d77-f7507ec6796a" volumeName="kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-service-ca-bundle" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075830    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151b0e6d-c5a6-469c-8d77-f7507ec6796a" volumeName="kubernetes.io/secret/151b0e6d-c5a6-469c-8d77-f7507ec6796a-serving-cert" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075846    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53ee9fcc-846d-4ca8-835f-58d77e3f1a63" volumeName="kubernetes.io/empty-dir/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-tmp-dir" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075858    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="545aca01-0ea8-4df2-b3bc-fa60f6f1c725" volumeName="kubernetes.io/projected/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-kube-api-access-5ssd9" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075872    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f9c06cb-46f7-440b-b395-a4c4e9a597cd" volumeName="kubernetes.io/configmap/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-config" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075884    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abf3688b-7862-4d8c-8e8f-03a791523655" volumeName="kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075897    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3418cdf-4a4a-426f-bdcc-140c84c48a5a" volumeName="kubernetes.io/projected/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kube-api-access-sl9zc" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075908    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee248b83-7c43-46fe-809c-69b61ca6254a" volumeName="kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-stats-auth" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075921    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfdf39cf-ae26-400e-b929-80512e786f5a" volumeName="kubernetes.io/projected/bfdf39cf-ae26-400e-b929-80512e786f5a-kube-api-access-tpqfx" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075933    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3946b8e3-670c-407b-ba35-01cd37890b2d" volumeName="kubernetes.io/configmap/3946b8e3-670c-407b-ba35-01cd37890b2d-nginx-conf" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075945    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="492e27da-7982-4035-945f-066588cfdb79" volumeName="kubernetes.io/configmap/492e27da-7982-4035-945f-066588cfdb79-iptables-alerter-script" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075958    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee248b83-7c43-46fe-809c-69b61ca6254a" volumeName="kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-default-certificate" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075974    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151b0e6d-c5a6-469c-8d77-f7507ec6796a" volumeName="kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075987    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46ca8995-d9ba-42fb-94b6-d7cf18696cce" volumeName="kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.075999    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="170428c7-02e4-40f5-a88c-1634d8b58b5a" volumeName="kubernetes.io/projected/170428c7-02e4-40f5-a88c-1634d8b58b5a-kube-api-access-8dpww" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076012    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46ca8995-d9ba-42fb-94b6-d7cf18696cce" volumeName="kubernetes.io/empty-dir/46ca8995-d9ba-42fb-94b6-d7cf18696cce-data-volume" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076116    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="492e27da-7982-4035-945f-066588cfdb79" volumeName="kubernetes.io/projected/492e27da-7982-4035-945f-066588cfdb79-kube-api-access-mmp24" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076131    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b434899-5102-48c6-a2cf-c37ccd545f8a" volumeName="kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-tuned" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076144    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88128ae2-7d2c-41a4-a0d1-ceb286a7d80f" volumeName="kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-config" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076177    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d8dc680-3f86-4ec8-949a-18cb41c680d1" volumeName="kubernetes.io/projected/9d8dc680-3f86-4ec8-949a-18cb41c680d1-kube-api-access-2xr94" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076189    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="170428c7-02e4-40f5-a88c-1634d8b58b5a" volumeName="kubernetes.io/secret/170428c7-02e4-40f5-a88c-1634d8b58b5a-serving-cert" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076201    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88128ae2-7d2c-41a4-a0d1-ceb286a7d80f" volumeName="kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076212    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88128ae2-7d2c-41a4-a0d1-ceb286a7d80f" volumeName="kubernetes.io/projected/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-kube-api-access-m2b2l" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076223    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4cabc49-e7ed-4c2a-8652-114a52f0f7a5" volumeName="kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-daemon-config" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076235    5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1113fe81-67a7-4c02-ae59-d9fd15e22bd0" volumeName="kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz" seLinuxMountContext=""
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076247    5196 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:29:20.077239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.076253    5196 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:29:20.084849 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.084818    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:29:20.084849 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.084840    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:29:20.092383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.092349    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:29:20.092539 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.092401    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:29:20.099141 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.099059    5196 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:29:20.100499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.100469    5196 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:29:20.100499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.100495    5196 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:29:20.100678 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.100520    5196 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:29:20.100678 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.100559    5196 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:29:20.121439 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.121390    5196 generic.go:356] "Generic (PLEG): container finished" podID="e77962c08d121df4200a38924d523dec" containerID="f3335c11d04ad0d5e7a2c7ce98866064a2922aff489d41c9a1af1978467a5fd1" exitCode=0
Nov 03 20:29:20.135032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.135001    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:29:20.135790 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.135747    5196 generic.go:356] "Generic (PLEG): container finished" podID="88128ae2-7d2c-41a4-a0d1-ceb286a7d80f" containerID="c752cfa451488c7cb10f22f24b9028101ed1dfb3203e0a9402ec6f8fb693cf07" exitCode=1
Nov 03 20:29:20.144735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.144707    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:29:20.144735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.144737    5196 generic.go:356] "Generic (PLEG): container finished" podID="170428c7-02e4-40f5-a88c-1634d8b58b5a" containerID="8921072839d44f3eea0077736b80615c3b142efc9f2f5d07700e5f58f72abfb2" exitCode=255
Nov 03 20:29:20.153992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.153956    5196 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="a7c8e7db7ac1b281626c79063947765b41935ad15887e7b9b2b608eb58b9d641" exitCode=0
Nov 03 20:29:20.153992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.153987    5196 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="78e53326524745322fe794262d8f9917d985529885ee07d37b13329617185cec" exitCode=0
Nov 03 20:29:20.153992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.153998    5196 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="3a326b7833effadf3e5e910972ce170538e89f8266be06462cf8fd600f6227d2" exitCode=0
Nov 03 20:29:20.154235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.154011    5196 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="96a7b84d3ad84fa8d1ff0486dae1e00f93fde30fa65fd6b44d26ba8cfdf34a1e" exitCode=0
Nov 03 20:29:20.154235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.154024    5196 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="2edb00fc77c43e22d753613239afd8e069d7986b48fab760212b772cb255e9b2" exitCode=0
Nov 03 20:29:20.154235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.154033    5196 generic.go:356] "Generic (PLEG): container finished" podID="abf3688b-7862-4d8c-8e8f-03a791523655" containerID="27264f31fbe7da242def105a6fc72e992aad72fc5f5d0561c899618de9f645c3" exitCode=0
Nov 03 20:29:20.166551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.166524    5196 manager.go:324] Recovery completed
Nov 03 20:29:20.170825 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.170802    5196 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-49.us-west-2.compute.internal\" not found"
Nov 03 20:29:20.183689 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.183667    5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:20.186195 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.186132    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:20.186195 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.186175    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:20.186195 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.186187    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:20.189486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189465    5196 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:29:20.189486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189482    5196 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:29:20.189486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189484    5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:20.189636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189501    5196 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:29:20.189665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189632    5196 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 20:29:20.189665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189643    5196 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 20:29:20.189665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189660    5196 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 20:29:20.189765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189667    5196 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 20:29:20.189765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189674    5196 policy_none.go:50] "Start"
Nov 03 20:29:20.189765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189686    5196 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:29:20.189765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.189696    5196 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:29:20.190509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.190492    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:20.190586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.190524    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:20.190586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.190534    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:20.191153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.191139    5196 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 20:29:20.191185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.191157    5196 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 20:29:20.191185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.191164    5196 policy_none.go:44] "Start"
Nov 03 20:29:20.195854 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.195807    5196 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:29:20.199964 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.199948    5196 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:29:20.200050 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.199974    5196 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:29:20.200050 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.199984    5196 server.go:87] "Starting device plugin registration server"
Nov 03 20:29:20.200343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200330    5196 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:29:20.200373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200346    5196 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:29:20.200448 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200431    5196 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:29:20.200578 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200542    5196 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:29:20.200618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200587    5196 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:29:20.200618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200593    5196 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:29:20.200703 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200667    5196 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"]
Nov 03 20:29:20.200753 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200648    5196 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:29:20.200575749Z","UUID":"2565dbc4-f81b-4363-9ba3-e3ae2c1aece1","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:29:20.200837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.200815    5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:20.202043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.202019    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:20.202185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.202062    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:20.202185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.202078    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:20.202480 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.202459    5196 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:29:20.202601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.202509    5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:20.202601 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.202523    5196 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-49.us-west-2.compute.internal\" not found"
Nov 03 20:29:20.202601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.202510    5196 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:29:20.202601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.202585    5196 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:29:20.202793 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.202643    5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:20.203540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203520    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:20.203636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203559    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:20.203636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203527    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:20.203636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203573    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:20.203636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203587    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:20.203636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203611    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:20.203865 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.203734    5196 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-49.us-west-2.compute.internal\" not found" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.203939 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203910    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"5b0fa7d4d6c53e4753bcd43d896ecfe1","Type":"ContainerStarted","Data":"ec5afee12ba6d33a9f6005389a77c44b572851e89f25d91bab32266920ef012b"}
Nov 03 20:29:20.203969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203950    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"5b0fa7d4d6c53e4753bcd43d896ecfe1","Type":"ContainerStarted","Data":"33e2c1bfb20b02194555408becd8026b73a838f919064c807c2790f11efa3df0"}
Nov 03 20:29:20.203998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203968    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"e77962c08d121df4200a38924d523dec","Type":"ContainerStarted","Data":"f5b10b1cdbbf287d467a1ec4bfbbbd8c432f76fa55a76f3160f298b39a75bbb2"}
Nov 03 20:29:20.203998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.203987    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"e77962c08d121df4200a38924d523dec","Type":"ContainerDied","Data":"f3335c11d04ad0d5e7a2c7ce98866064a2922aff489d41c9a1af1978467a5fd1"}
Nov 03 20:29:20.204056 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.204006    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal" event={"ID":"e77962c08d121df4200a38924d523dec","Type":"ContainerStarted","Data":"9ac08017bcf4ab7a41e54278af39a59868dc4eccf639011e5b8f457ac05b2b39"}
Nov 03 20:29:20.204056 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.204023    5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:20.204855 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.204828    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:20.204920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.204860    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:20.204920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.204874    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:20.205018 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.205002    5196 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-49.us-west-2.compute.internal\" not found" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.259852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.259812    5196 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:29:20.260027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.259886    5196 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 20:29:20.260027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.259943    5196 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:29:20.260184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.260058    5196 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:29:20.271036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.271010    5196 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.302423 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.302388    5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:20.303581 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.303563    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:20.303705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.303594    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:20.303705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.303604    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:20.303705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.303648    5196 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.307254 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.307229    5196 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:29:20.352479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.352449    5196 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:29:20.352619 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.352500    5196 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.352619 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.352516    5196 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.360392 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.360366    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.360519 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.360395    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.360519 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.360422    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0fa7d4d6c53e4753bcd43d896ecfe1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"5b0fa7d4d6c53e4753bcd43d896ecfe1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.406976 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.406952    5196 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.407079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.407020    5196 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.407602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.407589    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeNotReady"
Nov 03 20:29:20.407660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.407619    5196 setters.go:546] "Node became not ready" node="ip-10-0-128-49.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T20:29:20Z","lastTransitionTime":"2026-11-03T20:29:20Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 20:29:20.427655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.427627    5196 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:29:20.427778 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:20.427676    5196 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.457407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.457330    5196 nodeinfomanager.go:471] existing CSINode "ip-10-0-128-49.us-west-2.compute.internal" is owned by different node (oldNodeID="2da8d096-ad06-4ce2-ab13-0687e529814b", newNodeID=""), cleaning up...
Nov 03 20:29:20.461281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.461249    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0fa7d4d6c53e4753bcd43d896ecfe1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"5b0fa7d4d6c53e4753bcd43d896ecfe1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.461391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.461296    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.461391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.461320    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.461391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.461358    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0fa7d4d6c53e4753bcd43d896ecfe1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"5b0fa7d4d6c53e4753bcd43d896ecfe1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.461391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.461369    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.461530 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.461369    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77962c08d121df4200a38924d523dec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" (UID: \"e77962c08d121df4200a38924d523dec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:20.670886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.670856    5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-49.us-west-2.compute.internal" event="NodeReady"
Nov 03 20:29:20.671022 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:20.670934    5196 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:29:21.057477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.057426    5196 apiserver.go:52] "Watching apiserver"
Nov 03 20:29:21.155630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.155597    5196 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:29:21.156314 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.156286    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hpchn","openshift-ingress/router-default-c7454997c-ccrh5","kube-system/konnectivity-agent-bmhst","openshift-network-operator/iptables-alerter-qmz42","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal","openshift-multus/multus-8jhfn","openshift-console-operator/console-operator-7cd56b8dbc-v455q","openshift-insights/insights-operator-7c4986d449-g2rqz","openshift-cluster-node-tuning-operator/tuned-c6fhq","openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76","openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj","openshift-dns/dns-default-k9c8l","kube-system/kas-connection-checker-685895c7c9-qpwf2","kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf","openshift-multus/network-metrics-daemon-g67qg","openshift-image-registry/node-ca-nlnjm","kube-system/global-pull-secret-syncer-8w5qd","kube-system/kas-connection-checker-685895c7c9-6vt8w","openshift-dns/node-resolver-np7pp","openshift-insights/insights-runtime-extractor-lc4mt","openshift-network-console/networking-console-plugin-86c478bb88-6q6fg","openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9","openshift-multus/multus-additional-cni-plugins-x5kbn","openshift-network-diagnostics/network-check-target-tjt8g","openshift-ovn-kubernetes/ovnkube-node-shbqc","kube-system/kas-connection-checker-685895c7c9-hjc54"]
Nov 03 20:29:21.157105 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.157074    5196 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:21.157334 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.157314    5196 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:21.158155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.158140    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.158458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.158441    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.159432 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.159415    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.160356 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.160340    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:29:21.165135 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.165118    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:29:21.169380 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.169363    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.173424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.173408    5196 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:29:21.206602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.206576    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.206722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.206636    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.206722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.206653    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.206722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.206702    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.207797 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.207767    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 20:29:21.207905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.207841    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.207905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.207877    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.208016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.207943    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.208016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.207963    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.208175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208159    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 20:29:21.208257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208172    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 20:29:21.208257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208225    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.208257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208235    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 20:29:21.208257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208234    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 20:29:21.208447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208380    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 20:29:21.208496 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208479    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 20:29:21.208551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208516    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 20:29:21.208870 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.208856    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 20:29:21.214616 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.214602    5196 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:29:21.214669 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:21.214651    5196 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:21.215012 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.214999    5196 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:29:21.215044 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:21.215036    5196 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-49.us-west-2.compute.internal"
Nov 03 20:29:21.250817 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.250791    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 20:29:21.251160 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.251145    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 20:29:21.251591 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.251575    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 20:29:21.253243 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.253227    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 20:29:21.253346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.253271    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 20:29:21.253458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.253434    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-b585d\""
Nov 03 20:29:21.253515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.253489    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fpmzx\""
Nov 03 20:29:21.253567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.253536    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.253624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.253614    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.254032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254004    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:29:21.254078 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254034    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.254224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254196    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:29:21.254352 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254322    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.254352 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254353    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.254537 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254369    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 20:29:21.254537 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254508    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.254641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254535    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.254641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254549    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 20:29:21.254876 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.254862    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:29:21.255266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.255250    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:29:21.255445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.255427    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.255529 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.255432    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.255529 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.255469    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.255622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.255591    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jxjwz\""
Nov 03 20:29:21.255924 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.255891    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.255999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.255952    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:29:21.255999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.255963    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:29:21.256235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256220    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.256494 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256474    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.256601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256562    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.256601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256590    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:29:21.256705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256680    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 20:29:21.256748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256703    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.256906 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256891    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.256952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256894    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.256952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.256943    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8pktm\""
Nov 03 20:29:21.257069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257046    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.257309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257294    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:29:21.257350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257339    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:29:21.257383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257360    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:29:21.257383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257364    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.257383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257369    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:29:21.257561 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257549    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:29:21.257642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257622    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.257642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257640    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.257749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257689    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.257805 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257791    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 20:29:21.257888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257876    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pqh8h\""
Nov 03 20:29:21.257992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.257979    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntq9n\""
Nov 03 20:29:21.258150 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.258133    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d54s8\""
Nov 03 20:29:21.259666 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.259157    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mlgd9\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.260913    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9qmc\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.260961    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w48hg\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261030    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6vtd\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261229    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261302    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261369    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxhx8\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261445    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261500    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kpkvl\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261567    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h9ssl\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261612    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261703    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261776    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261841    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261931    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jb4cj\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261963    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.261963    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.262012    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.262049    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.262078    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:29:21.262175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.262136    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:29:21.262768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.262618    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6582\""
Nov 03 20:29:21.263105 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.263074    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 20:29:21.263149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.263079    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 20:29:21.265561 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265532    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.265614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265573    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-tmp\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.265614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265599    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpgb4\" (UniqueName: \"kubernetes.io/projected/1d9f21de-9ec8-48b7-aa66-5945a79d95e4-kube-api-access-rpgb4\") pod \"kas-connection-checker-685895c7c9-6vt8w\" (UID: \"1d9f21de-9ec8-48b7-aa66-5945a79d95e4\") " pod="kube-system/kas-connection-checker-685895c7c9-6vt8w"
Nov 03 20:29:21.265714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265696    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-etc-kubernetes\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.265752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265725    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-conf\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.265752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265742    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46ca8995-d9ba-42fb-94b6-d7cf18696cce-crio-socket\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.265806 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265700    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-tmp\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.265806 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265758    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.265806 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265777    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-env-overrides\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.265806 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265800    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.265918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265832    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:29:21.265999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.265983    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3b4704b-2627-4530-8100-838f2d8b7d31-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.266032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266012    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpqfx\" (UniqueName: \"kubernetes.io/projected/bfdf39cf-ae26-400e-b929-80512e786f5a-kube-api-access-tpqfx\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:29:21.266032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266030    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-multus\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.266107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266045    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-multus-certs\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.266107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266042    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-env-overrides\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.266107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266061    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d8dc680-3f86-4ec8-949a-18cb41c680d1-serviceca\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:29:21.266107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266077    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-modprobe-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.266253 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266108    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-host\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.266253 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266148    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492e27da-7982-4035-945f-066588cfdb79-host-slash\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:29:21.266253 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266183    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-service-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.266253 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266213    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovn-node-metrics-cert\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.266253 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266238    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b325b4-c286-465f-83c9-623e0d441d17-config\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:29:21.266253 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266241    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3b4704b-2627-4530-8100-838f2d8b7d31-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.266520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266281    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.266520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266387    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.266520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266422    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b325b4-c286-465f-83c9-623e0d441d17-config\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:29:21.266520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266441    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151b0e6d-c5a6-469c-8d77-f7507ec6796a-serving-cert\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.266520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266479    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlcpg\" (UniqueName: \"kubernetes.io/projected/151b0e6d-c5a6-469c-8d77-f7507ec6796a-kube-api-access-nlcpg\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.266520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266506    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssd9\" (UniqueName: \"kubernetes.io/projected/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-kube-api-access-5ssd9\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.266815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266525    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.266815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266530    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.266815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266558    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xr94\" (UniqueName: \"kubernetes.io/projected/9d8dc680-3f86-4ec8-949a-18cb41c680d1-kube-api-access-2xr94\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:29:21.266815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266585    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xkj\" (UniqueName: \"kubernetes.io/projected/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-kube-api-access-c8xkj\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.266815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266748    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b325b4-c286-465f-83c9-623e0d441d17-serving-cert\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:29:21.266815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266787    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-device-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266853    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266866    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d8dc680-3f86-4ec8-949a-18cb41c680d1-serviceca\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266887    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266906    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151b0e6d-c5a6-469c-8d77-f7507ec6796a-serving-cert\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266915    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-service-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266923    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-run\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.266979    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-netns\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267009    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovn-node-metrics-cert\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267024    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-var-lib-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267044    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b325b4-c286-465f-83c9-623e0d441d17-serving-cert\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:29:21.267077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267057    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-ovn\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267135    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267178    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-hosts-file\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267210    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtt5\" (UniqueName: \"kubernetes.io/projected/4b434899-5102-48c6-a2cf-c37ccd545f8a-kube-api-access-mmtt5\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267236    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwc28\" (UniqueName: \"kubernetes.io/projected/e8b325b4-c286-465f-83c9-623e0d441d17-kube-api-access-fwc28\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267254    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-config\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267321    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/170428c7-02e4-40f5-a88c-1634d8b58b5a-serving-cert\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267423    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-bin\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267433    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-config\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267443    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267488    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267505    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-systemd\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.267524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267519    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-daemon-config\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267593    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/170428c7-02e4-40f5-a88c-1634d8b58b5a-serving-cert\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267684    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvg9\" (UniqueName: \"kubernetes.io/projected/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a-kube-api-access-7wvg9\") pod \"volume-data-source-validator-58c9f5f46d-hswtf\" (UID: \"859d83db-cfcc-459f-8ed2-8ec6a7e88b1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267702    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267716    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-log-socket\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267723    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/428ef791-c562-4786-b095-7c3b1113bfc8-original-pull-secret\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267731    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-sys\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267778    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-kubelet\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267800    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-etc-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267805    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/151b0e6d-c5a6-469c-8d77-f7507ec6796a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267819    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpww\" (UniqueName: \"kubernetes.io/projected/170428c7-02e4-40f5-a88c-1634d8b58b5a-kube-api-access-8dpww\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267842    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267854    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-daemon-config\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267874    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-konnectivity-ca\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267926    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.267944    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-kubelet-config\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:21.268041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268040    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-config-volume\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268063    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-konnectivity-ca\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268078    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-netd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268216    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpl4\" (UniqueName: \"kubernetes.io/projected/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-kube-api-access-nvpl4\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268243    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-config-volume\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268251    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3946b8e3-670c-407b-ba35-01cd37890b2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268268    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkz4p\" (UniqueName: \"kubernetes.io/projected/68f3b7ee-a558-4c4d-8625-11f72d13b257-kube-api-access-qkz4p\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268290    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/492e27da-7982-4035-945f-066588cfdb79-iptables-alerter-script\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268410    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cnibin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268442    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-kubelet\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268456    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/492e27da-7982-4035-945f-066588cfdb79-iptables-alerter-script\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268458    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3946b8e3-670c-407b-ba35-01cd37890b2d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268465    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-conf-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268514    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268541    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268569    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-lib-modules\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268594    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmp24\" (UniqueName: \"kubernetes.io/projected/492e27da-7982-4035-945f-066588cfdb79-kube-api-access-mmp24\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268629    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-system-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268645    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-socket-dir-parent\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268699    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-snapshots\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268731    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-systemd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.268749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268758    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268807    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268814    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/151b0e6d-c5a6-469c-8d77-f7507ec6796a-snapshots\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268819    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-config\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268847    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-slash\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268893    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-tuned\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268924    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-bin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.268967    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-tuned\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269001    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcfn\" (UniqueName: \"kubernetes.io/projected/a3b4704b-2627-4530-8100-838f2d8b7d31-kube-api-access-fdcfn\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269018    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-config\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269021    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269148    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-dbus\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269184    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-sys-fs\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269213    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269243    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcz4p\" (UniqueName: \"kubernetes.io/projected/abf3688b-7862-4d8c-8e8f-03a791523655-kube-api-access-dcz4p\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269290    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-node-log\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269365    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2b2l\" (UniqueName: \"kubernetes.io/projected/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-kube-api-access-m2b2l\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269394    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269439    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269468    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-agent-certs\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269493    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca8995-d9ba-42fb-94b6-d7cf18696cce-data-volume\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269534    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-systemd-units\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.269551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269558    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-script-lib\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269584    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nx5\" (UniqueName: \"kubernetes.io/projected/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-api-access-46nx5\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269593    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269594    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca8995-d9ba-42fb-94b6-d7cf18696cce-data-volume\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269613    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-hostroot\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269736    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3-agent-certs\") pod \"konnectivity-agent-bmhst\" (UID: \"e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3\") " pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269788    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269807    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-ovnkube-script-lib\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269839    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-socket-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269879    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-registration-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269906    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-tmp-dir\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269965    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-tmp-dir\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.269990    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqk6q\" (UniqueName: \"kubernetes.io/projected/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-kube-api-access-mqk6q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270014    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270032    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-netns\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270197    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-k8s-cni-cncf-io\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270211    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46ca8995-d9ba-42fb-94b6-d7cf18696cce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270229    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9zc\" (UniqueName: \"kubernetes.io/projected/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kube-api-access-sl9zc\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270246    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8sqv\" (UniqueName: \"kubernetes.io/projected/ee248b83-7c43-46fe-809c-69b61ca6254a-kube-api-access-s8sqv\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.270302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270286    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-kubernetes\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270322    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270355    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt5pj\" (UniqueName: \"kubernetes.io/projected/68c418d8-0eb1-44a4-9c0f-59d3d7d724af-kube-api-access-xt5pj\") pod \"kas-connection-checker-685895c7c9-hjc54\" (UID: \"68c418d8-0eb1-44a4-9c0f-59d3d7d724af\") " pod="kube-system/kas-connection-checker-685895c7c9-hjc54"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270380    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270426    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-default-certificate\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270490    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-trusted-ca\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270545    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270582    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270610    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-var-lib-kubelet\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270634    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d8dc680-3f86-4ec8-949a-18cb41c680d1-host\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270653    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-default-certificate\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270658    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-cnibin\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270684    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-tmp-dir\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270722    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfdf39cf-ae26-400e-b929-80512e786f5a-cert\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270750    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysconfig\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270782    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmt7s\" (UniqueName: \"kubernetes.io/projected/17826854-67bc-4852-97c1-83ebe2c648f7-kube-api-access-wmt7s\") pod \"network-check-source-6f78d59f99-hpm76\" (UID: \"17826854-67bc-4852-97c1-83ebe2c648f7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270794    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270805    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/170428c7-02e4-40f5-a88c-1634d8b58b5a-trusted-ca\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270796    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abf3688b-7862-4d8c-8e8f-03a791523655-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270802    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-tmp-dir\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270812    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqjh\" (UniqueName: \"kubernetes.io/projected/a533118b-20c1-478a-b7e7-42dd7b4f200c-kube-api-access-szqjh\") pod \"kas-connection-checker-685895c7c9-qpwf2\" (UID: \"a533118b-20c1-478a-b7e7-42dd7b4f200c\") " pod="kube-system/kas-connection-checker-685895c7c9-qpwf2"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270852    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-stats-auth\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270882    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-system-cni-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270905    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-tmp\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270940    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cni-binary-copy\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270944    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfdf39cf-ae26-400e-b929-80512e786f5a-cert\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270970    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b434899-5102-48c6-a2cf-c37ccd545f8a-tmp\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.271013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.270970    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8gd\" (UniqueName: \"kubernetes.io/projected/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-kube-api-access-xp8gd\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:29:21.271737 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.271121    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cni-binary-copy\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.271737 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.271152    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-stats-auth\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.317765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.317687    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpgb4\" (UniqueName: \"kubernetes.io/projected/1d9f21de-9ec8-48b7-aa66-5945a79d95e4-kube-api-access-rpgb4\") pod \"kas-connection-checker-685895c7c9-6vt8w\" (UID: \"1d9f21de-9ec8-48b7-aa66-5945a79d95e4\") " pod="kube-system/kas-connection-checker-685895c7c9-6vt8w"
Nov 03 20:29:21.318059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.318034    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmqtz\" (UniqueName: \"kubernetes.io/projected/1113fe81-67a7-4c02-ae59-d9fd15e22bd0-kube-api-access-nmqtz\") pod \"network-check-target-tjt8g\" (UID: \"1113fe81-67a7-4c02-ae59-d9fd15e22bd0\") " pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:29:21.318147 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.318124    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpqfx\" (UniqueName: \"kubernetes.io/projected/bfdf39cf-ae26-400e-b929-80512e786f5a-kube-api-access-tpqfx\") pod \"ingress-canary-hpchn\" (UID: \"bfdf39cf-ae26-400e-b929-80512e786f5a\") " pod="openshift-ingress-canary/ingress-canary-hpchn"
Nov 03 20:29:21.318443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.318425    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xr94\" (UniqueName: \"kubernetes.io/projected/9d8dc680-3f86-4ec8-949a-18cb41c680d1-kube-api-access-2xr94\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:29:21.318567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.318538    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xkj\" (UniqueName: \"kubernetes.io/projected/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-kube-api-access-c8xkj\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.318956 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.318940    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssd9\" (UniqueName: \"kubernetes.io/projected/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-kube-api-access-5ssd9\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.319011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.318996    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlcpg\" (UniqueName: \"kubernetes.io/projected/151b0e6d-c5a6-469c-8d77-f7507ec6796a-kube-api-access-nlcpg\") pod \"insights-operator-7c4986d449-g2rqz\" (UID: \"151b0e6d-c5a6-469c-8d77-f7507ec6796a\") " pod="openshift-insights/insights-operator-7c4986d449-g2rqz"
Nov 03 20:29:21.350405 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.350375    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwc28\" (UniqueName: \"kubernetes.io/projected/e8b325b4-c286-465f-83c9-623e0d441d17-kube-api-access-fwc28\") pod \"service-ca-operator-544d749c8c-zwbpf\" (UID: \"e8b325b4-c286-465f-83c9-623e0d441d17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf"
Nov 03 20:29:21.352363 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.352330    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2b2l\" (UniqueName: \"kubernetes.io/projected/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-kube-api-access-m2b2l\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.353040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.352469    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmt7s\" (UniqueName: \"kubernetes.io/projected/17826854-67bc-4852-97c1-83ebe2c648f7-kube-api-access-wmt7s\") pod \"network-check-source-6f78d59f99-hpm76\" (UID: \"17826854-67bc-4852-97c1-83ebe2c648f7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hpm76"
Nov 03 20:29:21.353040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.352523    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkz4p\" (UniqueName: \"kubernetes.io/projected/68f3b7ee-a558-4c4d-8625-11f72d13b257-kube-api-access-qkz4p\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.353040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.352777    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpww\" (UniqueName: \"kubernetes.io/projected/170428c7-02e4-40f5-a88c-1634d8b58b5a-kube-api-access-8dpww\") pod \"console-operator-7cd56b8dbc-v455q\" (UID: \"170428c7-02e4-40f5-a88c-1634d8b58b5a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:21.353040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.352889    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqk6q\" (UniqueName: \"kubernetes.io/projected/9f9c06cb-46f7-440b-b395-a4c4e9a597cd-kube-api-access-mqk6q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-66vk9\" (UID: \"9f9c06cb-46f7-440b-b395-a4c4e9a597cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9"
Nov 03 20:29:21.353040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.352913    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtt5\" (UniqueName: \"kubernetes.io/projected/4b434899-5102-48c6-a2cf-c37ccd545f8a-kube-api-access-mmtt5\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.353040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.352981    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmp24\" (UniqueName: \"kubernetes.io/projected/492e27da-7982-4035-945f-066588cfdb79-kube-api-access-mmp24\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:29:21.353364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.353171    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcfn\" (UniqueName: \"kubernetes.io/projected/a3b4704b-2627-4530-8100-838f2d8b7d31-kube-api-access-fdcfn\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.353364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.353180    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvg9\" (UniqueName: \"kubernetes.io/projected/859d83db-cfcc-459f-8ed2-8ec6a7e88b1a-kube-api-access-7wvg9\") pod \"volume-data-source-validator-58c9f5f46d-hswtf\" (UID: \"859d83db-cfcc-459f-8ed2-8ec6a7e88b1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hswtf"
Nov 03 20:29:21.353364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.353210    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nx5\" (UniqueName: \"kubernetes.io/projected/46ca8995-d9ba-42fb-94b6-d7cf18696cce-kube-api-access-46nx5\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.353364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.353250    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt5pj\" (UniqueName: \"kubernetes.io/projected/68c418d8-0eb1-44a4-9c0f-59d3d7d724af-kube-api-access-xt5pj\") pod \"kas-connection-checker-685895c7c9-hjc54\" (UID: \"68c418d8-0eb1-44a4-9c0f-59d3d7d724af\") " pod="kube-system/kas-connection-checker-685895c7c9-hjc54"
Nov 03 20:29:21.353364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.353337    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpl4\" (UniqueName: \"kubernetes.io/projected/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-kube-api-access-nvpl4\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:29:21.353509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.353375    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8gd\" (UniqueName: \"kubernetes.io/projected/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-kube-api-access-xp8gd\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:29:21.353509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.353494    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcz4p\" (UniqueName: \"kubernetes.io/projected/abf3688b-7862-4d8c-8e8f-03a791523655-kube-api-access-dcz4p\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.353742 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.353725    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqjh\" (UniqueName: \"kubernetes.io/projected/a533118b-20c1-478a-b7e7-42dd7b4f200c-kube-api-access-szqjh\") pod \"kas-connection-checker-685895c7c9-qpwf2\" (UID: \"a533118b-20c1-478a-b7e7-42dd7b4f200c\") " pod="kube-system/kas-connection-checker-685895c7c9-qpwf2"
Nov 03 20:29:21.354369 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.354354    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9zc\" (UniqueName: \"kubernetes.io/projected/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kube-api-access-sl9zc\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.354553 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.354539    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8sqv\" (UniqueName: \"kubernetes.io/projected/ee248b83-7c43-46fe-809c-69b61ca6254a-kube-api-access-s8sqv\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.372173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372141    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-systemd-units\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.372232 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372178    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-hostroot\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372232 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372195    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.372295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372237    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-socket-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.372295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372260    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-registration-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.372295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372269    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-systemd-units\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.372295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372283    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-netns\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372269    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-hostroot\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372306    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372310    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-k8s-cni-cncf-io\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372326    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-socket-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372342    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-k8s-cni-cncf-io\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372355    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-kubernetes\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372374    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372377    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-registration-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372394    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372374    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-netns\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.372487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372434    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-var-lib-kubelet\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372488    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-var-lib-kubelet\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372517    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-kubernetes\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372509    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d8dc680-3f86-4ec8-949a-18cb41c680d1-host\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372540    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d8dc680-3f86-4ec8-949a-18cb41c680d1-host\") pod \"node-ca-nlnjm\" (UID: \"9d8dc680-3f86-4ec8-949a-18cb41c680d1\") " pod="openshift-image-registry/node-ca-nlnjm"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372551    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-cnibin\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372579    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysconfig\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372605    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-cnibin\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372607    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-system-cni-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372640    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-system-cni-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372654    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysconfig\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372655    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372680    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-etc-kubernetes\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372696    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-conf\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372715    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46ca8995-d9ba-42fb-94b6-d7cf18696cce-crio-socket\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372734    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-etc-kubernetes\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372744    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372744    5196 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372774    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372800    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-multus\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372824    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-multus-certs\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372830    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372848    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-conf\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372771    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46ca8995-d9ba-42fb-94b6-d7cf18696cce-crio-socket\") pod \"insights-runtime-extractor-lc4mt\" (UID: \"46ca8995-d9ba-42fb-94b6-d7cf18696cce\") " pod="openshift-insights/insights-runtime-extractor-lc4mt"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372851    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-modprobe-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372824    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372885    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-multus-certs\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372901    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-multus\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.372945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372939    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-host\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.372978    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492e27da-7982-4035-945f-066588cfdb79-host-slash\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373020    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-host\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373038    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/492e27da-7982-4035-945f-066588cfdb79-host-slash\") pod \"iptables-alerter-qmz42\" (UID: \"492e27da-7982-4035-945f-066588cfdb79\") " pod="openshift-network-operator/iptables-alerter-qmz42"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373065    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-modprobe-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373111    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373152    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373157    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373189    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-device-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373225    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373248    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373281    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-run\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373303    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-netns\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373312    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-run-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373327    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-var-lib-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373353    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-ovn\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373356    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-device-dir\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373357    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee248b83-7c43-46fe-809c-69b61ca6254a-service-ca-bundle\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373376    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373282    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-sysctl-d\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373400    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-hosts-file\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373436    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-bin\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373461    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-systemd\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373487    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-log-socket\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373510    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-sys\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373528    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-var-lib-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373530    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-kubelet\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373558    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-kubelet\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373615    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-sys\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373617    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-log-socket\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373593    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-etc-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373684    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373714    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-kubelet-config\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373690    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-run\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373742    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-netd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373750    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53ee9fcc-846d-4ca8-835f-58d77e3f1a63-hosts-file\") pod \"node-resolver-np7pp\" (UID: \"53ee9fcc-846d-4ca8-835f-58d77e3f1a63\") " pod="openshift-dns/node-resolver-np7pp"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373770    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cnibin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373769    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-kubelet-config\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373792    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-kubelet\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373815    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-netd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373818    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-etc-systemd\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373712    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-etc-openvswitch\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373820    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-conf-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373863    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373861    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-run-netns\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373887    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-lib-modules\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373894    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-cni-bin\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373912    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-kubelet\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373865    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-conf-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373941    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-system-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373946    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abf3688b-7862-4d8c-8e8f-03a791523655-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5kbn\" (UID: \"abf3688b-7862-4d8c-8e8f-03a791523655\") " pod="openshift-multus/multus-additional-cni-plugins-x5kbn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.373986    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-socket-dir-parent\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374014    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-systemd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374044    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b434899-5102-48c6-a2cf-c37ccd545f8a-lib-modules\") pod \"tuned-c6fhq\" (UID: \"4b434899-5102-48c6-a2cf-c37ccd545f8a\") " pod="openshift-cluster-node-tuning-operator/tuned-c6fhq"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374045    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-ovn\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374064    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374108    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-system-cni-dir\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374109    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-multus-socket-dir-parent\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374125    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-slash\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374159    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-run-systemd\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.374125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374162    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374196    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-bin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374211    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-cnibin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374227    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-host-slash\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374289    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374342    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4cabc49-e7ed-4c2a-8652-114a52f0f7a5-host-var-lib-cni-bin\") pod \"multus-8jhfn\" (UID: \"a4cabc49-e7ed-4c2a-8652-114a52f0f7a5\") " pod="openshift-multus/multus-8jhfn"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374364    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-dbus\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374491    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-sys-fs\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374662    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-node-log\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374723    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3418cdf-4a4a-426f-bdcc-140c84c48a5a-sys-fs\") pod \"aws-ebs-csi-driver-node-kdft4\" (UID: \"c3418cdf-4a4a-426f-bdcc-140c84c48a5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdft4"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374573    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/428ef791-c562-4786-b095-7c3b1113bfc8-dbus\") pod \"global-pull-secret-syncer-8w5qd\" (UID: \"428ef791-c562-4786-b095-7c3b1113bfc8\") " pod="kube-system/global-pull-secret-syncer-8w5qd"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.374799    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88128ae2-7d2c-41a4-a0d1-ceb286a7d80f-node-log\") pod \"ovnkube-node-shbqc\" (UID: \"88128ae2-7d2c-41a4-a0d1-ceb286a7d80f\") " pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.375886    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6-metrics-certs\") pod \"network-metrics-daemon-g67qg\" (UID: \"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6\") " pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.375896    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68f3b7ee-a558-4c4d-8625-11f72d13b257-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zd5zj\" (UID: \"68f3b7ee-a558-4c4d-8625-11f72d13b257\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.376177    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3946b8e3-670c-407b-ba35-01cd37890b2d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6q6fg\" (UID: \"3946b8e3-670c-407b-ba35-01cd37890b2d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.376407    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/545aca01-0ea8-4df2-b3bc-fa60f6f1c725-metrics-tls\") pod \"dns-default-k9c8l\" (UID: \"545aca01-0ea8-4df2-b3bc-fa60f6f1c725\") " pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.377145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.376799    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3b4704b-2627-4530-8100-838f2d8b7d31-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4g2kg\" (UID: \"a3b4704b-2627-4530-8100-838f2d8b7d31\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.377607 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.377468    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee248b83-7c43-46fe-809c-69b61ca6254a-metrics-certs\") pod \"router-default-c7454997c-ccrh5\" (UID: \"ee248b83-7c43-46fe-809c-69b61ca6254a\") " pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.458787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.458747    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:21.458787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.458767    5196 scope.go:122] "RemoveContainer" containerID="8921072839d44f3eea0077736b80615c3b142efc9f2f5d07700e5f58f72abfb2"
Nov 03 20:29:21.459026 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.458883    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"
Nov 03 20:29:21.459762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.459742    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"
Nov 03 20:29:21.460643 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.460625    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"
Nov 03 20:29:21.468544 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.465749    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g67qg"
Nov 03 20:29:21.469638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.469617    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:21.924207 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.924121    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg"]
Nov 03 20:29:21.932473 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:21.932443    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b4704b_2627_4530_8100_838f2d8b7d31.slice/crio-ac1f1e313075da7f05ea71c006e763f6d7c0e0f734864f305d088db8673b9e44 WatchSource:0}: Error finding container ac1f1e313075da7f05ea71c006e763f6d7c0e0f734864f305d088db8673b9e44: Status 404 returned error can't find the container with id ac1f1e313075da7f05ea71c006e763f6d7c0e0f734864f305d088db8673b9e44
Nov 03 20:29:21.934901 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.934868    5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:29:21.972886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.972855    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9c8l"]
Nov 03 20:29:21.975598 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.975571    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c7454997c-ccrh5"]
Nov 03 20:29:21.978989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.978966    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6q6fg"]
Nov 03 20:29:21.979647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.979558    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g67qg"]
Nov 03 20:29:21.979819 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.979779    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj"]
Nov 03 20:29:21.981086 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:21.981051    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545aca01_0ea8_4df2_b3bc_fa60f6f1c725.slice/crio-d8dcafffdfb385f30c03531683970b9e7713a053f4be2c74f4cf4f6453e17332 WatchSource:0}: Error finding container d8dcafffdfb385f30c03531683970b9e7713a053f4be2c74f4cf4f6453e17332: Status 404 returned error can't find the container with id d8dcafffdfb385f30c03531683970b9e7713a053f4be2c74f4cf4f6453e17332
Nov 03 20:29:21.981866 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:21.981793    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tjt8g"
Nov 03 20:29:21.983666 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:21.983646    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee248b83_7c43_46fe_809c_69b61ca6254a.slice/crio-b1f1a3d5f5e6c49438c0a342ff0517ffbca7e64d7459bec55c3761d55ed91b43 WatchSource:0}: Error finding container b1f1a3d5f5e6c49438c0a342ff0517ffbca7e64d7459bec55c3761d55ed91b43: Status 404 returned error can't find the container with id b1f1a3d5f5e6c49438c0a342ff0517ffbca7e64d7459bec55c3761d55ed91b43
Nov 03 20:29:22.027468 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:22.027435    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b2cede_7e06_4f9c_ac4b_ed1533ff4aa6.slice/crio-eaf3c3dd4f46b0b147ab7aec4c90a50dc652e61927f19094dfaabec377630de5 WatchSource:0}: Error finding container eaf3c3dd4f46b0b147ab7aec4c90a50dc652e61927f19094dfaabec377630de5: Status 404 returned error can't find the container with id eaf3c3dd4f46b0b147ab7aec4c90a50dc652e61927f19094dfaabec377630de5
Nov 03 20:29:22.028301 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:22.028268    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3946b8e3_670c_407b_ba35_01cd37890b2d.slice/crio-443f6356b824421f72a81288335e9259d05a6daf680e0df292099f873292ef32 WatchSource:0}: Error finding container 443f6356b824421f72a81288335e9259d05a6daf680e0df292099f873292ef32: Status 404 returned error can't find the container with id 443f6356b824421f72a81288335e9259d05a6daf680e0df292099f873292ef32
Nov 03 20:29:22.163187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.163162    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:29:22.163614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.163247    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" event={"ID":"170428c7-02e4-40f5-a88c-1634d8b58b5a","Type":"ContainerStarted","Data":"61bbb2100be7abb487bb65aa7c2322ca343ff0d56ef768fd49ce8cfd5ed14f20"}
Nov 03 20:29:22.164539 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.164515    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g67qg" event={"ID":"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6","Type":"ContainerStarted","Data":"eaf3c3dd4f46b0b147ab7aec4c90a50dc652e61927f19094dfaabec377630de5"}
Nov 03 20:29:22.165721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.165691    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg" event={"ID":"a3b4704b-2627-4530-8100-838f2d8b7d31","Type":"ContainerStarted","Data":"ac1f1e313075da7f05ea71c006e763f6d7c0e0f734864f305d088db8673b9e44"}
Nov 03 20:29:22.167118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.167072    5196 generic.go:356] "Generic (PLEG): container finished" podID="151b0e6d-c5a6-469c-8d77-f7507ec6796a" containerID="c6c55f51279080c35dc5b48f87e81e9104827a28872283bfeea7e6ec3fc1e5e1" exitCode=0
Nov 03 20:29:22.167208 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.167147    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-g2rqz" event={"ID":"151b0e6d-c5a6-469c-8d77-f7507ec6796a","Type":"ContainerDied","Data":"c6c55f51279080c35dc5b48f87e81e9104827a28872283bfeea7e6ec3fc1e5e1"}
Nov 03 20:29:22.167460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.167441    5196 scope.go:122] "RemoveContainer" containerID="c6c55f51279080c35dc5b48f87e81e9104827a28872283bfeea7e6ec3fc1e5e1"
Nov 03 20:29:22.168397 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.168365    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg" event={"ID":"3946b8e3-670c-407b-ba35-01cd37890b2d","Type":"ContainerStarted","Data":"443f6356b824421f72a81288335e9259d05a6daf680e0df292099f873292ef32"}
Nov 03 20:29:22.169999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.169984    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c7454997c-ccrh5" event={"ID":"ee248b83-7c43-46fe-809c-69b61ca6254a","Type":"ContainerStarted","Data":"23c2aa7ac135e2f6cf4425fa4500534e56d8dd027162244048f5cb62c9ca2b36"}
Nov 03 20:29:22.170040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.170019    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c7454997c-ccrh5" event={"ID":"ee248b83-7c43-46fe-809c-69b61ca6254a","Type":"ContainerStarted","Data":"b1f1a3d5f5e6c49438c0a342ff0517ffbca7e64d7459bec55c3761d55ed91b43"}
Nov 03 20:29:22.171667 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.171646    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9c8l" event={"ID":"545aca01-0ea8-4df2-b3bc-fa60f6f1c725","Type":"ContainerStarted","Data":"d8dcafffdfb385f30c03531683970b9e7713a053f4be2c74f4cf4f6453e17332"}
Nov 03 20:29:22.173035 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.173007    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj" event={"ID":"68f3b7ee-a558-4c4d-8625-11f72d13b257","Type":"ContainerStarted","Data":"602df230578f82ebdf741f197088ca60601a744b64f448296ad42490a460f5d0"}
Nov 03 20:29:22.460036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.459959    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:22.462294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.462274    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:22.955681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.955643    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:22.956979 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:22.956953    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c7454997c-ccrh5"
Nov 03 20:29:23.177468 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:23.177317    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:23.179808 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:23.179775    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-g2rqz" event={"ID":"151b0e6d-c5a6-469c-8d77-f7507ec6796a","Type":"ContainerStarted","Data":"ce3664bb84bb913f3892c868fd2129afa8eef53a2623548c8853a5fded70130f"}
Nov 03 20:29:23.181613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:23.181588    5196 generic.go:356] "Generic (PLEG): container finished" podID="9f9c06cb-46f7-440b-b395-a4c4e9a597cd" containerID="e12f02b204568606b23b8b1c571653794a5d0f6e278ea7c1a5a298f2cedba7a2" exitCode=0
Nov 03 20:29:23.181721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:23.181676    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9" event={"ID":"9f9c06cb-46f7-440b-b395-a4c4e9a597cd","Type":"ContainerDied","Data":"e12f02b204568606b23b8b1c571653794a5d0f6e278ea7c1a5a298f2cedba7a2"}
Nov 03 20:29:23.182138 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:23.182121    5196 scope.go:122] "RemoveContainer" containerID="e12f02b204568606b23b8b1c571653794a5d0f6e278ea7c1a5a298f2cedba7a2"
Nov 03 20:29:23.381197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:23.381160    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:23.553068 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:23.552983    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:23.909434 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:23.909399    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:24.118148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:24.118084    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:24.188850 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:24.188772    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-66vk9" event={"ID":"9f9c06cb-46f7-440b-b395-a4c4e9a597cd","Type":"ContainerStarted","Data":"92b4d87c5b8bd426c7f5897e209499b05691f2827a7cef82f4d9ac95847ded3c"}
Nov 03 20:29:24.190450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:24.190425    5196 generic.go:356] "Generic (PLEG): container finished" podID="e8b325b4-c286-465f-83c9-623e0d441d17" containerID="4c6264356a8bcdc67869ccc397c7f8f97d398d6adb84872ef592d156d84170c3" exitCode=0
Nov 03 20:29:24.190567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:24.190492    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf" event={"ID":"e8b325b4-c286-465f-83c9-623e0d441d17","Type":"ContainerDied","Data":"4c6264356a8bcdc67869ccc397c7f8f97d398d6adb84872ef592d156d84170c3"}
Nov 03 20:29:24.190890 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:24.190873    5196 scope.go:122] "RemoveContainer" containerID="4c6264356a8bcdc67869ccc397c7f8f97d398d6adb84872ef592d156d84170c3"
Nov 03 20:29:24.268191 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:24.268155    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:25.198498 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:25.198460    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zwbpf" event={"ID":"e8b325b4-c286-465f-83c9-623e0d441d17","Type":"ContainerStarted","Data":"7d2bb660322cfc9930cd943e368247c3f84923ae2603cc1f6b77af3bfde8aa5f"}
Nov 03 20:29:25.853105 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:25.853056    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:26.034448 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:26.034412    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:26.687265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:26.687232    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:26.687638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:26.687428    5196 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:29:27.210149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:27.210112    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g67qg" event={"ID":"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6","Type":"ContainerStarted","Data":"4afddcae76f68c328b7a2b11ba9c72fa4f0cd7e2e5b15f469c2c4410c2ba1530"}
Nov 03 20:29:27.210346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:27.210179    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g67qg" event={"ID":"e5b2cede-7e06-4f9c-ac4b-ed1533ff4aa6","Type":"ContainerStarted","Data":"deac153cdb08bf6a7638ebf5a8f3cc2691be994053afa9f9da162d71e649e907"}
Nov 03 20:29:27.212300 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:27.212270    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4g2kg" event={"ID":"a3b4704b-2627-4530-8100-838f2d8b7d31","Type":"ContainerStarted","Data":"e9e534bef1600e8d73742cd88c55ce72326c5edf577849948dd2086598999c4d"}
Nov 03 20:29:27.213871 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:27.213847    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6q6fg" event={"ID":"3946b8e3-670c-407b-ba35-01cd37890b2d","Type":"ContainerStarted","Data":"b85b9a9d9567b2c3c1f88e78482d3ad449dafd1fce3872e735bac3ba41c00279"}
Nov 03 20:29:27.420832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:27.420796    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-np7pp_53ee9fcc-846d-4ca8-835f-58d77e3f1a63/dns-node-resolver/0.log"
Nov 03 20:29:27.762754 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:27.762725    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:27.912695 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:27.912661    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:28.065392 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.065310    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:28.219808 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.219765    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj" event={"ID":"68f3b7ee-a558-4c4d-8625-11f72d13b257","Type":"ContainerStarted","Data":"cd5272930c49b216575197f9856b9e69c14bc81e8f75b26210eaaf5e489ceef6"}
Nov 03 20:29:28.219990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.219830    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zd5zj" event={"ID":"68f3b7ee-a558-4c4d-8625-11f72d13b257","Type":"ContainerStarted","Data":"2f18fe44473b28c17edcc242263b03421b90ff5c44051004bf3320cd67d7cfbf"}
Nov 03 20:29:28.420550 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.420522    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nlnjm_9d8dc680-3f86-4ec8-949a-18cb41c680d1/node-ca/0.log"
Nov 03 20:29:28.621034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.621001    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7454997c-ccrh5_ee248b83-7c43-46fe-809c-69b61ca6254a/router/0.log"
Nov 03 20:29:28.757660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.757574    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:28.757820 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.757722    5196 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:29:28.761484 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.761460    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q"
Nov 03 20:29:28.858902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:28.858864    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:29.020819 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:29.020747    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hpchn_bfdf39cf-ae26-400e-b929-80512e786f5a/serve-healthcheck-canary/0.log"
Nov 03 20:29:29.043845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:29.043815    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:29.617408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:29.617370    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:29.785824 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:29.785791    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:30.314486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:30.314449    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:32.236166 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:32.236133    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9c8l" event={"ID":"545aca01-0ea8-4df2-b3bc-fa60f6f1c725","Type":"ContainerStarted","Data":"016d04dde1607952be95d1b7ae05ceee9da941b52090b403375268f97a6bf698"}
Nov 03 20:29:32.236562 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:32.236184    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9c8l" event={"ID":"545aca01-0ea8-4df2-b3bc-fa60f6f1c725","Type":"ContainerStarted","Data":"5367aa05b3a74b7818b2974d8f3132f1a217cdffec676117d79da1a3a8dd21c5"}
Nov 03 20:29:32.236562 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:32.236268    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:33.142146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:33.142113    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:33.342969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:33.342933    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:33.490935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:33.490849    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:33.640698 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:33.640646    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:34.198229 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:34.198195    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:42.245600 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:42.245567    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k9c8l"
Nov 03 20:29:42.463991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:42.463953    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:42.607655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:42.607620    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:43.663887 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:43.663846    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:44.383968 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:44.383931    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:44.545943 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:44.545905    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:44.695209 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:44.695075    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:44.849999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:44.849957    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:45.010895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.010814    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-796cf8fc4-v6f98"]
Nov 03 20:29:45.014235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.014214    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.019458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.019428    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:45.063779 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.063743    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-service-ca\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.063779 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.063794    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-serving-cert\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.064019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.063814    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-config\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.064019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.063936    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-trusted-ca-bundle\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.064019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.063998    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31c0bfe1-9744-43c4-86ae-b6542ac63988-tmp\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.064180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.064050    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-oauth-config\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.064180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.064071    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-oauth-serving-cert\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.064180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.064116    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gd5\" (UniqueName: \"kubernetes.io/projected/31c0bfe1-9744-43c4-86ae-b6542ac63988-kube-api-access-98gd5\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.074407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.074366    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:29:45.077765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.077740    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:29:45.122368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.121507    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p55vv\""
Nov 03 20:29:45.122368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.121565    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:29:45.122368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.121603    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:29:45.122368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.121609    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:29:45.122368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.121698    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:29:45.122368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.121749    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:29:45.125059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.124779    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shbqc"
Nov 03 20:29:45.165500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.165470    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-oauth-config\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.165500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.165503    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-oauth-serving-cert\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.165762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.165522    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gd5\" (UniqueName: \"kubernetes.io/projected/31c0bfe1-9744-43c4-86ae-b6542ac63988-kube-api-access-98gd5\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.165762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.165542    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-service-ca\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.165762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.165598    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-serving-cert\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.165762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.165621    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-config\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.165762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.165665    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-trusted-ca-bundle\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.165762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.165742    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31c0bfe1-9744-43c4-86ae-b6542ac63988-tmp\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.166134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.166079    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31c0bfe1-9744-43c4-86ae-b6542ac63988-tmp\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.166325 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.166303    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-oauth-serving-cert\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.166420 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.166399    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-service-ca\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.166726 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.166642    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-config\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.168506 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.168477    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-oauth-config\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.168924 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.168905    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-serving-cert\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.188912 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.188872    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:45.216458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.216428    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:29:45.217233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.217211    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796cf8fc4-v6f98"]
Nov 03 20:29:45.223044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.223014    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gd5\" (UniqueName: \"kubernetes.io/projected/31c0bfe1-9744-43c4-86ae-b6542ac63988-kube-api-access-98gd5\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.227516 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.227484    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-trusted-ca-bundle\") pod \"console-796cf8fc4-v6f98\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") " pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.257431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.257401    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gcl5k"]
Nov 03 20:29:45.260933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.260869    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.323988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.323958    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:29:45.324198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.324179    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:29:45.324290 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.324266    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2c9vt\""
Nov 03 20:29:45.326433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.326408    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:45.355659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.355609    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:45.367414 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367377    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.367604 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367457    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f61a647-3229-4c6e-b181-a44abbea314e-metrics-client-ca\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.367669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367608    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-root\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.367731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367666    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.367731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367706    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-tls\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.367845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367779    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-wtmp\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.367845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367821    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-sys\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.367951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367850    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-textfile\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.367951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.367892    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl9rc\" (UniqueName: \"kubernetes.io/projected/2f61a647-3229-4c6e-b181-a44abbea314e-kube-api-access-hl9rc\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.400117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.397638    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:29:45.400117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.397940    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:29:45.468889 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.468857    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-wtmp\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.468889 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.468894    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-sys\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.468914    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-textfile\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.468990    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-sys\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469031    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl9rc\" (UniqueName: \"kubernetes.io/projected/2f61a647-3229-4c6e-b181-a44abbea314e-kube-api-access-hl9rc\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469048    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-wtmp\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469108    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469185    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f61a647-3229-4c6e-b181-a44abbea314e-metrics-client-ca\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469217    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-root\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469228    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-textfile\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469245    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469282    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-tls\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469287    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f61a647-3229-4c6e-b181-a44abbea314e-root\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469863    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.469918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.469896    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f61a647-3229-4c6e-b181-a44abbea314e-metrics-client-ca\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.471668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.471638    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-tls\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.471782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.471675    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f61a647-3229-4c6e-b181-a44abbea314e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.507880 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.507830    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:45.525772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.525681    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl9rc\" (UniqueName: \"kubernetes.io/projected/2f61a647-3229-4c6e-b181-a44abbea314e-kube-api-access-hl9rc\") pod \"node-exporter-gcl5k\" (UID: \"2f61a647-3229-4c6e-b181-a44abbea314e\") " pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.575629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.575597    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gcl5k"
Nov 03 20:29:45.587593 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:45.587557    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f61a647_3229_4c6e_b181_a44abbea314e.slice/crio-cf3af209255e7f4b4b85ec568e105cf2d59bf16326095cf6af8808483c4e331e WatchSource:0}: Error finding container cf3af209255e7f4b4b85ec568e105cf2d59bf16326095cf6af8808483c4e331e: Status 404 returned error can't find the container with id cf3af209255e7f4b4b85ec568e105cf2d59bf16326095cf6af8808483c4e331e
Nov 03 20:29:45.666980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.666938    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:45.728524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.728490    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796cf8fc4-v6f98"]
Nov 03 20:29:45.738326 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:45.738287    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c0bfe1_9744_43c4_86ae_b6542ac63988.slice/crio-04e3d0a8f854aa38e8af55fc1a42d0b3cda049c3bd4621a9b1f25cf4a521e0e9 WatchSource:0}: Error finding container 04e3d0a8f854aa38e8af55fc1a42d0b3cda049c3bd4621a9b1f25cf4a521e0e9: Status 404 returned error can't find the container with id 04e3d0a8f854aa38e8af55fc1a42d0b3cda049c3bd4621a9b1f25cf4a521e0e9
Nov 03 20:29:45.903852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:45.903822    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:46.291729 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:46.291643    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796cf8fc4-v6f98" event={"ID":"31c0bfe1-9744-43c4-86ae-b6542ac63988","Type":"ContainerStarted","Data":"04e3d0a8f854aa38e8af55fc1a42d0b3cda049c3bd4621a9b1f25cf4a521e0e9"}
Nov 03 20:29:46.292911 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:46.292885    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcl5k" event={"ID":"2f61a647-3229-4c6e-b181-a44abbea314e","Type":"ContainerStarted","Data":"cf3af209255e7f4b4b85ec568e105cf2d59bf16326095cf6af8808483c4e331e"}
Nov 03 20:29:46.417926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:46.417889    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:46.673921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:46.673885    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:47.133485 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.133440    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:47.285828 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.285796    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:47.805808 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.805712    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"]
Nov 03 20:29:47.809621 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.809587    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.818268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.818231    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"]
Nov 03 20:29:47.863990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.863932    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 20:29:47.863990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.863959    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 20:29:47.863990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.863963    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 20:29:47.863990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.863988    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 20:29:47.864379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.864024    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 20:29:47.864379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.864147    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5rhcbbsqtss6g\""
Nov 03 20:29:47.864379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.864240    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s5w8k\""
Nov 03 20:29:47.891613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.891575    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.891782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.891629    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e732ae-04e9-4ae2-83da-3664f681018c-tmp\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.891782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.891723    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.891782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.891767    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e732ae-04e9-4ae2-83da-3664f681018c-metrics-client-ca\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.891955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.891836    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-grpc-tls\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.891955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.891858    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.891955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.891935    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td789\" (UniqueName: \"kubernetes.io/projected/06e732ae-04e9-4ae2-83da-3664f681018c-kube-api-access-td789\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.892131 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.891966    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-tls\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.892131 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.892003    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.911972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.911938    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bmhst"
Nov 03 20:29:47.993180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993141    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993185    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e732ae-04e9-4ae2-83da-3664f681018c-metrics-client-ca\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993224    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-grpc-tls\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993255    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993293    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td789\" (UniqueName: \"kubernetes.io/projected/06e732ae-04e9-4ae2-83da-3664f681018c-kube-api-access-td789\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993341    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-tls\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993386    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993474    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993515    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e732ae-04e9-4ae2-83da-3664f681018c-tmp\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.993868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.993841    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e732ae-04e9-4ae2-83da-3664f681018c-tmp\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.994584 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.994016    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e732ae-04e9-4ae2-83da-3664f681018c-metrics-client-ca\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.996983 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.996946    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.996983 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.996974    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-tls\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.997127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.997002    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-grpc-tls\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.997127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.997081    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.997522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.997500    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:47.997563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:47.997527    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06e732ae-04e9-4ae2-83da-3664f681018c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:48.048873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:48.048830    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td789\" (UniqueName: \"kubernetes.io/projected/06e732ae-04e9-4ae2-83da-3664f681018c-kube-api-access-td789\") pod \"thanos-querier-6cf8575ffd-g9cmq\" (UID: \"06e732ae-04e9-4ae2-83da-3664f681018c\") " pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:48.132074 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:48.132039    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:48.306125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:48.305750    5196 generic.go:356] "Generic (PLEG): container finished" podID="2f61a647-3229-4c6e-b181-a44abbea314e" containerID="8d19000d776babc339445fe7e5b4b584c810fc8feda91b820c82134f7b17a1f4" exitCode=0
Nov 03 20:29:48.306125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:48.305813    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcl5k" event={"ID":"2f61a647-3229-4c6e-b181-a44abbea314e","Type":"ContainerDied","Data":"8d19000d776babc339445fe7e5b4b584c810fc8feda91b820c82134f7b17a1f4"}
Nov 03 20:29:48.490783 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:29:48.490703    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:29:48.583072 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:48.583034    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"]
Nov 03 20:29:48.590323 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:48.590293    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e732ae_04e9_4ae2_83da_3664f681018c.slice/crio-5c3370be5aa09549d3f1728412dda48a55667a8b954c7eddd03afec70638674b WatchSource:0}: Error finding container 5c3370be5aa09549d3f1728412dda48a55667a8b954c7eddd03afec70638674b: Status 404 returned error can't find the container with id 5c3370be5aa09549d3f1728412dda48a55667a8b954c7eddd03afec70638674b
Nov 03 20:29:49.311451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.311363    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796cf8fc4-v6f98" event={"ID":"31c0bfe1-9744-43c4-86ae-b6542ac63988","Type":"ContainerStarted","Data":"db4a8992c4ef68d998cb77fafc791b7b997fdf51f921ca58f0a6658376efb59a"}
Nov 03 20:29:49.312730 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.312702    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq" event={"ID":"06e732ae-04e9-4ae2-83da-3664f681018c","Type":"ContainerStarted","Data":"5c3370be5aa09549d3f1728412dda48a55667a8b954c7eddd03afec70638674b"}
Nov 03 20:29:49.314902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.314877    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcl5k" event={"ID":"2f61a647-3229-4c6e-b181-a44abbea314e","Type":"ContainerStarted","Data":"b7234c04ba5009e976a2d51bc28e58bbae6e1a3be8b5718ccb108eb5c2dc108b"}
Nov 03 20:29:49.315024 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.314921    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcl5k" event={"ID":"2f61a647-3229-4c6e-b181-a44abbea314e","Type":"ContainerStarted","Data":"c815899823e8c7f4af64be6c0252c2de140d96cd5bc09899eddcbe3dd71f409d"}
Nov 03 20:29:49.319170 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.319130    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:49.419834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.419736    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-796cf8fc4-v6f98" podStartSLOduration=2.10038529 podStartE2EDuration="5.419715693s" totalImagesPullingTime="3.319330403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:49.419703276 +0000 UTC m=+29.423662021" watchObservedRunningTime="2026-11-03 20:29:49.419715693 +0000 UTC m=+29.423674418"
Nov 03 20:29:49.525656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.525616    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:49.529224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.529174    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gcl5k" podStartSLOduration=2.749928169 podStartE2EDuration="4.5291587s" totalImagesPullingTime="1.686551105s" totalInitContainerRuntime="92.679426ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:49.528920793 +0000 UTC m=+29.532879516" watchObservedRunningTime="2026-11-03 20:29:49.5291587 +0000 UTC m=+29.533117419"
Nov 03 20:29:49.671565 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.671531    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:49.825490 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.825455    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:49.825662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.825585    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"]
Nov 03 20:29:49.839553 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.839530    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:49.881355 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.881308    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"]
Nov 03 20:29:49.888790 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.888762    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zsfvq\""
Nov 03 20:29:49.888925 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.888846    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 20:29:49.909229 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.909189    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cgr\" (UniqueName: \"kubernetes.io/projected/bf1dbf60-0a31-48f3-aa2b-6d32468e6054-kube-api-access-j9cgr\") pod \"monitoring-plugin-77f4f76594-mlgwh\" (UID: \"bf1dbf60-0a31-48f3-aa2b-6d32468e6054\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:49.909373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.909249    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf1dbf60-0a31-48f3-aa2b-6d32468e6054-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mlgwh\" (UID: \"bf1dbf60-0a31-48f3-aa2b-6d32468e6054\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:49.972522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:49.972438    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:50.010246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.010213    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cgr\" (UniqueName: \"kubernetes.io/projected/bf1dbf60-0a31-48f3-aa2b-6d32468e6054-kube-api-access-j9cgr\") pod \"monitoring-plugin-77f4f76594-mlgwh\" (UID: \"bf1dbf60-0a31-48f3-aa2b-6d32468e6054\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:50.010462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.010262    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf1dbf60-0a31-48f3-aa2b-6d32468e6054-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mlgwh\" (UID: \"bf1dbf60-0a31-48f3-aa2b-6d32468e6054\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:50.012968 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.012942    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf1dbf60-0a31-48f3-aa2b-6d32468e6054-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mlgwh\" (UID: \"bf1dbf60-0a31-48f3-aa2b-6d32468e6054\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:50.064118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.064069    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cgr\" (UniqueName: \"kubernetes.io/projected/bf1dbf60-0a31-48f3-aa2b-6d32468e6054-kube-api-access-j9cgr\") pod \"monitoring-plugin-77f4f76594-mlgwh\" (UID: \"bf1dbf60-0a31-48f3-aa2b-6d32468e6054\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:50.162083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.162049    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:50.163414 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.163388    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6947bdbb66-klklq"]
Nov 03 20:29:50.182402 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.182369    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.223189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.222818    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6947bdbb66-klklq"]
Nov 03 20:29:50.232224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.232191    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:29:50.232416 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.232249    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 20:29:50.232525 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.232490    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 20:29:50.232600 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.232590    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gw8vc\""
Nov 03 20:29:50.232644 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.232595    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 20:29:50.232692 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.232657    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6ctqs8jv1d43i\""
Nov 03 20:29:50.317406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.317372    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-secret-metrics-server-tls\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.317903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.317412    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e697a4b7-ab69-445d-9354-4e8ad77ef608-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.317903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.317487    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfflz\" (UniqueName: \"kubernetes.io/projected/e697a4b7-ab69-445d-9354-4e8ad77ef608-kube-api-access-rfflz\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.317903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.317528    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e697a4b7-ab69-445d-9354-4e8ad77ef608-metrics-server-audit-profiles\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.317903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.317555    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-client-ca-bundle\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.317903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.317605    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-secret-metrics-server-client-certs\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.317903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.317651    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e697a4b7-ab69-445d-9354-4e8ad77ef608-audit-log\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.418204 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.418163    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-secret-metrics-server-tls\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.418375 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.418217    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e697a4b7-ab69-445d-9354-4e8ad77ef608-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.418375 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.418346    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfflz\" (UniqueName: \"kubernetes.io/projected/e697a4b7-ab69-445d-9354-4e8ad77ef608-kube-api-access-rfflz\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.418477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.418459    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e697a4b7-ab69-445d-9354-4e8ad77ef608-metrics-server-audit-profiles\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.418531 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.418491    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-client-ca-bundle\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.418584 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.418532    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-secret-metrics-server-client-certs\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.418641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.418626    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e697a4b7-ab69-445d-9354-4e8ad77ef608-audit-log\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.418955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.418920    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e697a4b7-ab69-445d-9354-4e8ad77ef608-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.419219 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.419200    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e697a4b7-ab69-445d-9354-4e8ad77ef608-audit-log\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.419900 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.419848    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e697a4b7-ab69-445d-9354-4e8ad77ef608-metrics-server-audit-profiles\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.421325 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.421284    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-secret-metrics-server-tls\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.421550 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.421533    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-client-ca-bundle\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.421955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.421940    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e697a4b7-ab69-445d-9354-4e8ad77ef608-secret-metrics-server-client-certs\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.474028 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.473954    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfflz\" (UniqueName: \"kubernetes.io/projected/e697a4b7-ab69-445d-9354-4e8ad77ef608-kube-api-access-rfflz\") pod \"metrics-server-6947bdbb66-klklq\" (UID: \"e697a4b7-ab69-445d-9354-4e8ad77ef608\") " pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.543933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.543895    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:29:50.556903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:50.556869    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"]
Nov 03 20:29:50.564358 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:50.564325    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1dbf60_0a31_48f3_aa2b_6d32468e6054.slice/crio-d32d05cd9ad3a31ac3a21ae7936064071a90705b721209296916c7b36e8c6405 WatchSource:0}: Error finding container d32d05cd9ad3a31ac3a21ae7936064071a90705b721209296916c7b36e8c6405: Status 404 returned error can't find the container with id d32d05cd9ad3a31ac3a21ae7936064071a90705b721209296916c7b36e8c6405
Nov 03 20:29:51.002829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:51.002792    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6947bdbb66-klklq"]
Nov 03 20:29:51.009813 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:29:51.009769    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode697a4b7_ab69_445d_9354_4e8ad77ef608.slice/crio-169898c8b825a32c1bcd97e5763f6062152c8ee13bcc1dd5cae33a57cacd159a WatchSource:0}: Error finding container 169898c8b825a32c1bcd97e5763f6062152c8ee13bcc1dd5cae33a57cacd159a: Status 404 returned error can't find the container with id 169898c8b825a32c1bcd97e5763f6062152c8ee13bcc1dd5cae33a57cacd159a
Nov 03 20:29:51.333818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:51.333762    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh" event={"ID":"bf1dbf60-0a31-48f3-aa2b-6d32468e6054","Type":"ContainerStarted","Data":"d32d05cd9ad3a31ac3a21ae7936064071a90705b721209296916c7b36e8c6405"}
Nov 03 20:29:51.335508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:51.335471    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq" event={"ID":"e697a4b7-ab69-445d-9354-4e8ad77ef608","Type":"ContainerStarted","Data":"169898c8b825a32c1bcd97e5763f6062152c8ee13bcc1dd5cae33a57cacd159a"}
Nov 03 20:29:51.935966 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:51.935927    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:52.087882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:52.087820    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:52.238653 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:52.238564    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:52.343742 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:52.343705    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq" event={"ID":"06e732ae-04e9-4ae2-83da-3664f681018c","Type":"ContainerStarted","Data":"668f3e86b4ea91c8b9cad1ce4d4ba0960af5bc5ff25e0658428325113466b91c"}
Nov 03 20:29:52.344246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:52.343777    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq" event={"ID":"06e732ae-04e9-4ae2-83da-3664f681018c","Type":"ContainerStarted","Data":"1334d47a6bbbee9486c4bb9a1f7fce49baadb8c2012624615ae513588b33b4b9"}
Nov 03 20:29:52.344246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:52.343809    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq" event={"ID":"06e732ae-04e9-4ae2-83da-3664f681018c","Type":"ContainerStarted","Data":"a46cdde0475e9c5dc947d21ecfa7fd80162c6d7db04e9992034a407ac6ef0e08"}
Nov 03 20:29:53.350969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:53.350882    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh" event={"ID":"bf1dbf60-0a31-48f3-aa2b-6d32468e6054","Type":"ContainerStarted","Data":"34ff9809cf0841c265c8290715a519ab5af43520116a432f4365d49cbeb1cb1a"}
Nov 03 20:29:53.350969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:53.350947    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:53.352925 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:53.352883    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq" event={"ID":"e697a4b7-ab69-445d-9354-4e8ad77ef608","Type":"ContainerStarted","Data":"d111fa25b5e4038bd794d7b2e46837d309c6e327344a83b313e41ca3241c1f86"}
Nov 03 20:29:53.356895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:53.356870    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh"
Nov 03 20:29:53.458512 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:53.458443    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mlgwh" podStartSLOduration=2.383451056 podStartE2EDuration="4.458423399s" totalImagesPullingTime="2.074972343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:53.458337003 +0000 UTC m=+33.462295728" watchObservedRunningTime="2026-11-03 20:29:53.458423399 +0000 UTC m=+33.462382125"
Nov 03 20:29:53.676926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:53.676870    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq" podStartSLOduration=1.5988419550000001 podStartE2EDuration="3.676851692s" totalImagesPullingTime="2.078009737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:53.675937975 +0000 UTC m=+33.679896727" watchObservedRunningTime="2026-11-03 20:29:53.676851692 +0000 UTC m=+33.680810416"
Nov 03 20:29:54.205081 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:54.205046    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:54.360541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:54.360503    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq" event={"ID":"06e732ae-04e9-4ae2-83da-3664f681018c","Type":"ContainerStarted","Data":"6e60c322818bc751692558d93a811138c5e8ec39496b27f47a848ea20499ab97"}
Nov 03 20:29:54.360999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:54.360572    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq" event={"ID":"06e732ae-04e9-4ae2-83da-3664f681018c","Type":"ContainerStarted","Data":"da8817ab6fc50956062d8a06d05324a0a5c5de03ead2f8f222be696650022c6f"}
Nov 03 20:29:54.360999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:54.360598    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq" event={"ID":"06e732ae-04e9-4ae2-83da-3664f681018c","Type":"ContainerStarted","Data":"7203f8d84791ccd6ffbe03d1f4df2c37f038d87d205d02e42cb39f079dc99f73"}
Nov 03 20:29:54.520975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:54.520864    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq" podStartSLOduration=2.681081893 podStartE2EDuration="7.520851453s" totalImagesPullingTime="4.83976956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:54.474063193 +0000 UTC m=+34.478021919" watchObservedRunningTime="2026-11-03 20:29:54.520851453 +0000 UTC m=+34.524810174"
Nov 03 20:29:55.326830 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:55.326791    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:55.326993 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:55.326858    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:55.331674 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:55.331649    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:55.365739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:55.365709    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:29:55.368949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:55.368930    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:29:57.988216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:57.988180    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:58.142772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:58.142739    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:59.327112 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:59.327054    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:59.496894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:59.496859    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:59.690081 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:29:59.689986    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:00.735860 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:00.735797    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:01.375894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:01.375858    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6cf8575ffd-g9cmq"
Nov 03 20:30:02.120870 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:02.120835    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:02.271765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:02.271718    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:08.345799 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:08.345759    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796cf8fc4-v6f98"]
Nov 03 20:30:08.786918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:08.786822    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:08.948833 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:08.948779    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:09.172420 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:09.172386    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:09.327724 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:09.327684    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:10.545014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:10.544976    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:30:10.545453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:10.545075    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:30:14.618271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:14.618235    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:14.767881 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:14.767845    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:18.483525 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:30:18.483487    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:30:29.924971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:29.924929    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:30.550141 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:30.550104    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:30:30.553969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:30.553942    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6947bdbb66-klklq"
Nov 03 20:30:31.736603 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:31.736574    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:32.349985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:32.349950    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:33.366317 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.366271    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-796cf8fc4-v6f98" podUID="31c0bfe1-9744-43c4-86ae-b6542ac63988" containerName="console" containerID="cri-o://db4a8992c4ef68d998cb77fafc791b7b997fdf51f921ca58f0a6658376efb59a" gracePeriod=15
Nov 03 20:30:33.524528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.524497    5196 generic.go:356] "Generic (PLEG): container finished" podID="31c0bfe1-9744-43c4-86ae-b6542ac63988" containerID="db4a8992c4ef68d998cb77fafc791b7b997fdf51f921ca58f0a6658376efb59a" exitCode=0
Nov 03 20:30:33.524696 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.524574    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796cf8fc4-v6f98" event={"ID":"31c0bfe1-9744-43c4-86ae-b6542ac63988","Type":"ContainerDied","Data":"db4a8992c4ef68d998cb77fafc791b7b997fdf51f921ca58f0a6658376efb59a"}
Nov 03 20:30:33.615685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.615661    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:30:33.678631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678543    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-service-ca\") pod \"31c0bfe1-9744-43c4-86ae-b6542ac63988\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") "
Nov 03 20:30:33.678631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678600    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-oauth-serving-cert\") pod \"31c0bfe1-9744-43c4-86ae-b6542ac63988\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") "
Nov 03 20:30:33.678841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678632    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c0bfe1-9744-43c4-86ae-b6542ac63988-tmp\" (UniqueName: \"kubernetes.io/empty-dir/31c0bfe1-9744-43c4-86ae-b6542ac63988-tmp\") pod \"31c0bfe1-9744-43c4-86ae-b6542ac63988\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") "
Nov 03 20:30:33.678841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678736    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-serving-cert\") pod \"31c0bfe1-9744-43c4-86ae-b6542ac63988\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") "
Nov 03 20:30:33.678841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678791    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-trusted-ca-bundle\") pod \"31c0bfe1-9744-43c4-86ae-b6542ac63988\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") "
Nov 03 20:30:33.679000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678861    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c0bfe1-9744-43c4-86ae-b6542ac63988-kube-api-access-98gd5\" (UniqueName: \"kubernetes.io/projected/31c0bfe1-9744-43c4-86ae-b6542ac63988-kube-api-access-98gd5\") pod \"31c0bfe1-9744-43c4-86ae-b6542ac63988\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") "
Nov 03 20:30:33.679000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678888    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-config\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-config\") pod \"31c0bfe1-9744-43c4-86ae-b6542ac63988\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") "
Nov 03 20:30:33.679000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678904    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-oauth-config\") pod \"31c0bfe1-9744-43c4-86ae-b6542ac63988\" (UID: \"31c0bfe1-9744-43c4-86ae-b6542ac63988\") "
Nov 03 20:30:33.679000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.678899    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c0bfe1-9744-43c4-86ae-b6542ac63988-tmp" pod "31c0bfe1-9744-43c4-86ae-b6542ac63988" (UID: "31c0bfe1-9744-43c4-86ae-b6542ac63988"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:33.679229 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.679021    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-service-ca" pod "31c0bfe1-9744-43c4-86ae-b6542ac63988" (UID: "31c0bfe1-9744-43c4-86ae-b6542ac63988"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:33.679229 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.679201    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-oauth-serving-cert" pod "31c0bfe1-9744-43c4-86ae-b6542ac63988" (UID: "31c0bfe1-9744-43c4-86ae-b6542ac63988"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:33.679319 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.679260    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-trusted-ca-bundle" pod "31c0bfe1-9744-43c4-86ae-b6542ac63988" (UID: "31c0bfe1-9744-43c4-86ae-b6542ac63988"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:33.679319 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.679294    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-config" pod "31c0bfe1-9744-43c4-86ae-b6542ac63988" (UID: "31c0bfe1-9744-43c4-86ae-b6542ac63988"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:33.679398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.679348    5196 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-oauth-serving-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:33.679398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.679367    5196 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31c0bfe1-9744-43c4-86ae-b6542ac63988-tmp\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:33.679398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.679388    5196 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-trusted-ca-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:33.679485 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.679404    5196 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:33.681388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.681356    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-oauth-config" pod "31c0bfe1-9744-43c4-86ae-b6542ac63988" (UID: "31c0bfe1-9744-43c4-86ae-b6542ac63988"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:33.681388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.681374    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-serving-cert" pod "31c0bfe1-9744-43c4-86ae-b6542ac63988" (UID: "31c0bfe1-9744-43c4-86ae-b6542ac63988"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:33.681388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.681376    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c0bfe1-9744-43c4-86ae-b6542ac63988-kube-api-access-98gd5" pod "31c0bfe1-9744-43c4-86ae-b6542ac63988" (UID: "31c0bfe1-9744-43c4-86ae-b6542ac63988"). InnerVolumeSpecName "kube-api-access-98gd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:33.780655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.780609    5196 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-serving-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:33.780655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.780643    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98gd5\" (UniqueName: \"kubernetes.io/projected/31c0bfe1-9744-43c4-86ae-b6542ac63988-kube-api-access-98gd5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:33.780655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.780657    5196 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-config\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:33.780655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:33.780666    5196 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31c0bfe1-9744-43c4-86ae-b6542ac63988-console-oauth-config\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:34.530466 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:34.530383    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796cf8fc4-v6f98"
Nov 03 20:30:34.530866 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:34.530377    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796cf8fc4-v6f98" event={"ID":"31c0bfe1-9744-43c4-86ae-b6542ac63988","Type":"ContainerDied","Data":"04e3d0a8f854aa38e8af55fc1a42d0b3cda049c3bd4621a9b1f25cf4a521e0e9"}
Nov 03 20:30:34.530866 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:34.530525    5196 scope.go:122] "RemoveContainer" containerID="db4a8992c4ef68d998cb77fafc791b7b997fdf51f921ca58f0a6658376efb59a"
Nov 03 20:30:34.686155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:34.686116    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796cf8fc4-v6f98"]
Nov 03 20:30:34.732839 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:34.732805    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-796cf8fc4-v6f98"]
Nov 03 20:30:35.443850 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:35.443820    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:36.106010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:36.105975    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c0bfe1-9744-43c4-86ae-b6542ac63988" path="/var/lib/kubelet/pods/31c0bfe1-9744-43c4-86ae-b6542ac63988/volumes"
Nov 03 20:30:36.167795 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:36.167765    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:36.339837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:36.339793    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:48.483262 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:30:48.483219    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:30:53.203710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:53.203670    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:53.349242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:30:53.349211    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:04.060580 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:04.060494    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:04.251295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:04.251259    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:16.633279 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:16.633241    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:16.784933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:16.784895    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:18.483907 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:31:18.483868    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:31:23.375274 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.375232    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-546df65468-jgfh6"]
Nov 03 20:31:23.375772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.375728    5196 cpu_manager.go:414] "containerMap: removing container" podUID="31c0bfe1-9744-43c4-86ae-b6542ac63988" containerName="console"
Nov 03 20:31:23.375772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.375748    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c0bfe1-9744-43c4-86ae-b6542ac63988" containerName="console"
Nov 03 20:31:23.375893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.375832    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c0bfe1-9744-43c4-86ae-b6542ac63988" containerName="console"
Nov 03 20:31:23.378763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.378742    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.428473 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.428427    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:31:23.471656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.471624    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-serving-cert\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.471841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.471659    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-service-ca\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.471841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.471696    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-oauth-serving-cert\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.471841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.471777    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-config\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.471841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.471798    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-trusted-ca-bundle\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.471841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.471822    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-oauth-config\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.471841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.471836    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-tmp\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.472101 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.471915    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhrl\" (UniqueName: \"kubernetes.io/projected/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-kube-api-access-xfhrl\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.475467 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.475432    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:31:23.475637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.475474    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:31:23.475637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.475472    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p55vv\""
Nov 03 20:31:23.475637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.475519    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:31:23.475637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.475475    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:31:23.475637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.475593    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:31:23.475637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.475602    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:31:23.570747 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.570713    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:31:23.571907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.571886    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546df65468-jgfh6"]
Nov 03 20:31:23.572280 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572260    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-config\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.572330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572294    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-trusted-ca-bundle\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.572364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572327    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-oauth-config\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.572364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572348    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-tmp\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.572453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572377    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhrl\" (UniqueName: \"kubernetes.io/projected/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-kube-api-access-xfhrl\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.572453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572423    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-serving-cert\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.572611 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572533    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-service-ca\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.572611 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572599    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-oauth-serving-cert\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.572927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.572896    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-tmp\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.573198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.573172    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-config\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.573381 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.573349    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-oauth-serving-cert\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.573455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.573390    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-service-ca\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.573509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.573460    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-trusted-ca-bundle\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.575105 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.575068    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-oauth-config\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.575216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.575199    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-console-serving-cert\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.580785 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.580756    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:23.626554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.626468    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhrl\" (UniqueName: \"kubernetes.io/projected/017b2b73-4a58-43c9-a7ec-ec36cae10ba9-kube-api-access-xfhrl\") pod \"console-546df65468-jgfh6\" (UID: \"017b2b73-4a58-43c9-a7ec-ec36cae10ba9\") " pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.689836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.689802    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:23.731771 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:23.731739    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:24.058835 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:24.058746    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546df65468-jgfh6"]
Nov 03 20:31:24.066632 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:31:24.066598    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017b2b73_4a58_43c9_a7ec_ec36cae10ba9.slice/crio-b4be523ece1cfb583e6cf31a1c5e1479be838c3a9bcbfc2e51e3fc91e566c0f8 WatchSource:0}: Error finding container b4be523ece1cfb583e6cf31a1c5e1479be838c3a9bcbfc2e51e3fc91e566c0f8: Status 404 returned error can't find the container with id b4be523ece1cfb583e6cf31a1c5e1479be838c3a9bcbfc2e51e3fc91e566c0f8
Nov 03 20:31:24.732119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:24.732065    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546df65468-jgfh6" event={"ID":"017b2b73-4a58-43c9-a7ec-ec36cae10ba9","Type":"ContainerStarted","Data":"6cd8e448306c98daf9188d31aab849c25ce4513b6a7fff9c14ea02ea32a09de4"}
Nov 03 20:31:24.732510 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:24.732181    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546df65468-jgfh6" event={"ID":"017b2b73-4a58-43c9-a7ec-ec36cae10ba9","Type":"ContainerStarted","Data":"b4be523ece1cfb583e6cf31a1c5e1479be838c3a9bcbfc2e51e3fc91e566c0f8"}
Nov 03 20:31:24.841192 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:24.841134    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-546df65468-jgfh6" podStartSLOduration=1.841119597 podStartE2EDuration="1.841119597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:24.840900827 +0000 UTC m=+124.844859550" watchObservedRunningTime="2026-11-03 20:31:24.841119597 +0000 UTC m=+124.845078321"
Nov 03 20:31:24.939663 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:24.939623    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:25.131561 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:25.131522    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:29.624885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:29.624855    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:29.774119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:29.774058    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:33.690429 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:33.690393    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:33.690822 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:33.690458    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:33.695296 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:33.695275    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:33.782016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:33.781987    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546df65468-jgfh6"
Nov 03 20:31:36.513016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:31:36.512979    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:48.483753 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:31:48.483714    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:32:09.237178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:32:09.237142    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:18.483476 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:32:18.483437    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:32:36.840289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:32:36.840249    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:48.483411 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:32:48.483367    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:33:18.483836 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:33:18.483800    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:33:37.174347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:33:37.174308    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:48.483752 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:33:48.483715    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:34:18.483405 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:34:18.483325    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:34:20.123360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:20.123325    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:34:20.123870 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:20.123749    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:34:20.130025 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:20.129996    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:34:20.130223 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:20.130155    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:34:37.516458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:37.516415    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:46.673342 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:46.673305    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:48.483911 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:34:48.483871    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:34:49.626486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:49.626455    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:49.774847 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:49.774812    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:51.537386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:51.537335    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:51.728347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:51.728292    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:52.861492 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:52.861462    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:53.051898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.051863    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:53.552537 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.552498    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"]
Nov 03 20:34:53.556113 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.556065    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.628859 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.628826    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.629014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.628868    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.629014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.628988    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ff6\" (UniqueName: \"kubernetes.io/projected/7f8142b6-7bcc-480f-b049-280fa3fcd205-kube-api-access-b4ff6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.655232 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.655202    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 20:34:53.655232 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.655217    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 20:34:53.655669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.655653    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-zqvl9\""
Nov 03 20:34:53.656231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.656202    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"]
Nov 03 20:34:53.656918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.656901    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"]
Nov 03 20:34:53.660293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.660278    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.707721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.707689    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"]
Nov 03 20:34:53.730182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.730147    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.730370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.730246    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ff6\" (UniqueName: \"kubernetes.io/projected/7f8142b6-7bcc-480f-b049-280fa3fcd205-kube-api-access-b4ff6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.730370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.730290    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.730370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.730324    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.730541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.730441    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.730541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.730482    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vlm\" (UniqueName: \"kubernetes.io/projected/a6144e77-a0f7-4020-84a3-24fd5d08733f-kube-api-access-f9vlm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.730624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.730601    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.730712 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.730697    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.752060 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.752027    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"]
Nov 03 20:34:53.752907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.752888    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-2hhmp\""
Nov 03 20:34:53.755691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.755673    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.785201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.785174    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ff6\" (UniqueName: \"kubernetes.io/projected/7f8142b6-7bcc-480f-b049-280fa3fcd205-kube-api-access-b4ff6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.805185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.805119    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-whd2z\""
Nov 03 20:34:53.811453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.811422    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"]
Nov 03 20:34:53.831736 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.831705    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.831883 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.831740    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.831883 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.831776    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.831883 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.831837    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcsx\" (UniqueName: \"kubernetes.io/projected/53b45b22-3c6c-4c12-8800-2b8b8be5d358-kube-api-access-jmcsx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.831984 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.831925    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.831984 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.831954    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vlm\" (UniqueName: \"kubernetes.io/projected/a6144e77-a0f7-4020-84a3-24fd5d08733f-kube-api-access-f9vlm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.832206 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.832189    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.832252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.832224    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.861443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.861403    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"]
Nov 03 20:34:53.864573 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.864540    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:53.865039 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.865020    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:53.871395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.871370    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:34:53.887125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.887081    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vlm\" (UniqueName: \"kubernetes.io/projected/a6144e77-a0f7-4020-84a3-24fd5d08733f-kube-api-access-f9vlm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.914764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.914738    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-6ncd8\""
Nov 03 20:34:53.920568 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.920527    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"]
Nov 03 20:34:53.932753 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.932721    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwmg\" (UniqueName: \"kubernetes.io/projected/354e8583-212f-49bd-9bb0-118431b5d3f0-kube-api-access-5dwmg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:53.932905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.932782    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.932905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.932850    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.933021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.932902    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcsx\" (UniqueName: \"kubernetes.io/projected/53b45b22-3c6c-4c12-8800-2b8b8be5d358-kube-api-access-jmcsx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.933021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.932942    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:53.933021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.932988    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:53.933302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.933277    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.933382 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.933325    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:53.975881 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.975841    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:34:53.989325 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:53.989301    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcsx\" (UniqueName: \"kubernetes.io/projected/53b45b22-3c6c-4c12-8800-2b8b8be5d358-kube-api-access-jmcsx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:54.033827 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.033795    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:54.033975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.033869    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:54.033975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.033916    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwmg\" (UniqueName: \"kubernetes.io/projected/354e8583-212f-49bd-9bb0-118431b5d3f0-kube-api-access-5dwmg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:54.034829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.034462    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:54.034947 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.034840    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:54.071431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.071404    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:34:54.090631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.090606    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwmg\" (UniqueName: \"kubernetes.io/projected/354e8583-212f-49bd-9bb0-118431b5d3f0-kube-api-access-5dwmg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:54.182432 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.182396    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:34:54.243064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.243018    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"]
Nov 03 20:34:54.249159 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:34:54.249127    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8142b6_7bcc_480f_b049_280fa3fcd205.slice/crio-e8c88b9fc232812bb7eb1c20617751a7a883e7575e26082f74ef8da2f4029eae WatchSource:0}: Error finding container e8c88b9fc232812bb7eb1c20617751a7a883e7575e26082f74ef8da2f4029eae: Status 404 returned error can't find the container with id e8c88b9fc232812bb7eb1c20617751a7a883e7575e26082f74ef8da2f4029eae
Nov 03 20:34:54.251248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.251226    5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:34:54.346966 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.346882    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"]
Nov 03 20:34:54.353280 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:34:54.353253    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6144e77_a0f7_4020_84a3_24fd5d08733f.slice/crio-a13daaabbdfdb854a1fe23ecfc61e93982b75c06df2c5c88aa07603f0fbd0a9f WatchSource:0}: Error finding container a13daaabbdfdb854a1fe23ecfc61e93982b75c06df2c5c88aa07603f0fbd0a9f: Status 404 returned error can't find the container with id a13daaabbdfdb854a1fe23ecfc61e93982b75c06df2c5c88aa07603f0fbd0a9f
Nov 03 20:34:54.439045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.439011    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"]
Nov 03 20:34:54.448903 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:34:54.448873    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b45b22_3c6c_4c12_8800_2b8b8be5d358.slice/crio-2077ca5d4b21667504e52718a319f9306d7e19cb0ed84d231678e1438f584661 WatchSource:0}: Error finding container 2077ca5d4b21667504e52718a319f9306d7e19cb0ed84d231678e1438f584661: Status 404 returned error can't find the container with id 2077ca5d4b21667504e52718a319f9306d7e19cb0ed84d231678e1438f584661
Nov 03 20:34:54.548553 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.548523    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"]
Nov 03 20:34:54.555651 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:34:54.555623    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354e8583_212f_49bd_9bb0_118431b5d3f0.slice/crio-3335652e685feb38570f257c918c0c9f9de627754c821436133eac8fe5063853 WatchSource:0}: Error finding container 3335652e685feb38570f257c918c0c9f9de627754c821436133eac8fe5063853: Status 404 returned error can't find the container with id 3335652e685feb38570f257c918c0c9f9de627754c821436133eac8fe5063853
Nov 03 20:34:54.606008 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.605935    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k" event={"ID":"7f8142b6-7bcc-480f-b049-280fa3fcd205","Type":"ContainerStarted","Data":"e8c88b9fc232812bb7eb1c20617751a7a883e7575e26082f74ef8da2f4029eae"}
Nov 03 20:34:54.607122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.607082    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp" event={"ID":"354e8583-212f-49bd-9bb0-118431b5d3f0","Type":"ContainerStarted","Data":"3335652e685feb38570f257c918c0c9f9de627754c821436133eac8fe5063853"}
Nov 03 20:34:54.608250 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.608230    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm" event={"ID":"53b45b22-3c6c-4c12-8800-2b8b8be5d358","Type":"ContainerStarted","Data":"2077ca5d4b21667504e52718a319f9306d7e19cb0ed84d231678e1438f584661"}
Nov 03 20:34:54.609370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.609345    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m" event={"ID":"a6144e77-a0f7-4020-84a3-24fd5d08733f","Type":"ContainerStarted","Data":"a13daaabbdfdb854a1fe23ecfc61e93982b75c06df2c5c88aa07603f0fbd0a9f"}
Nov 03 20:34:54.894192 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.894116    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"]
Nov 03 20:34:54.900707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.898566    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:54.950023 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.949992    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gkmgv\""
Nov 03 20:34:54.951259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:54.951231    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"]
Nov 03 20:34:55.041839 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.041807    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.041988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.041864    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.042055 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.041992    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm64h\" (UniqueName: \"kubernetes.io/projected/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-kube-api-access-nm64h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.143445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.143407    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.143638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.143488    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm64h\" (UniqueName: \"kubernetes.io/projected/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-kube-api-access-nm64h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.143638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.143542    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.143827 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.143803    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.143903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.143886    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.197501 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.197423    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm64h\" (UniqueName: \"kubernetes.io/projected/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-kube-api-access-nm64h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.214252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.214219    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:34:55.585440 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.585404    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"]
Nov 03 20:34:55.604033 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:34:55.603996    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94825d4_98c3_4c4b_8dd2_de5d61b6f224.slice/crio-5882ed4bdf50b25aaf19fdbe0508ea16827ca37073b965edf1a7a458f50320e3 WatchSource:0}: Error finding container 5882ed4bdf50b25aaf19fdbe0508ea16827ca37073b965edf1a7a458f50320e3: Status 404 returned error can't find the container with id 5882ed4bdf50b25aaf19fdbe0508ea16827ca37073b965edf1a7a458f50320e3
Nov 03 20:34:55.616286 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:55.616255    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25" event={"ID":"c94825d4-98c3-4c4b-8dd2-de5d61b6f224","Type":"ContainerStarted","Data":"5882ed4bdf50b25aaf19fdbe0508ea16827ca37073b965edf1a7a458f50320e3"}
Nov 03 20:34:56.114581 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.114518    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"]
Nov 03 20:34:56.119199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.119159    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.172576 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.172543    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"]
Nov 03 20:34:56.216659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.216608    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-vhkxj\""
Nov 03 20:34:56.254755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.254478    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdx78\" (UniqueName: \"kubernetes.io/projected/b2138c4d-ff75-49d4-8ff4-94498408efd3-kube-api-access-jdx78\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.254755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.254573    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.254755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.254609    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.355150 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.355108    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.355337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.355163    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.355337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.355263    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdx78\" (UniqueName: \"kubernetes.io/projected/b2138c4d-ff75-49d4-8ff4-94498408efd3-kube-api-access-jdx78\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.356033 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.356007    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.356319 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.356285    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.420940 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.420851    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdx78\" (UniqueName: \"kubernetes.io/projected/b2138c4d-ff75-49d4-8ff4-94498408efd3-kube-api-access-jdx78\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.439345 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.439311    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:34:56.869122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:56.869067    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"]
Nov 03 20:34:56.891016 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:34:56.890977    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2138c4d_ff75_49d4_8ff4_94498408efd3.slice/crio-cddae8953e59c36afbb817dc87c7cfb151302f336207b4b00a970952f794df08 WatchSource:0}: Error finding container cddae8953e59c36afbb817dc87c7cfb151302f336207b4b00a970952f794df08: Status 404 returned error can't find the container with id cddae8953e59c36afbb817dc87c7cfb151302f336207b4b00a970952f794df08
Nov 03 20:34:57.633131 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:57.633070    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs" event={"ID":"b2138c4d-ff75-49d4-8ff4-94498408efd3","Type":"ContainerStarted","Data":"cddae8953e59c36afbb817dc87c7cfb151302f336207b4b00a970952f794df08"}
Nov 03 20:34:58.785523 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.783057    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"]
Nov 03 20:34:58.787567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.787542    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:58.839481 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.839417    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-chl6n\""
Nov 03 20:34:58.841642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.841603    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"]
Nov 03 20:34:58.882271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.881891    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:58.882271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.881953    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:58.882271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.882043    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdx2\" (UniqueName: \"kubernetes.io/projected/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-kube-api-access-9mdx2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:58.984981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.983616    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:58.984981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.983677    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:58.984981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.983764    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdx2\" (UniqueName: \"kubernetes.io/projected/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-kube-api-access-9mdx2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:58.984981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.984624    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:58.984981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:58.984907    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:59.050192 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:59.050120    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdx2\" (UniqueName: \"kubernetes.io/projected/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-kube-api-access-9mdx2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:59.110691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:59.110398    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:34:59.554959 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:59.554812    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"]
Nov 03 20:34:59.564384 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:34:59.564322    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d0fa35_02fe_48a0_a2b2_84c48fe7a2ef.slice/crio-a509f13d33e744d36eddfe9b60e1580eb67a7ea53dfc0e6258e7e262ab5dab08 WatchSource:0}: Error finding container a509f13d33e744d36eddfe9b60e1580eb67a7ea53dfc0e6258e7e262ab5dab08: Status 404 returned error can't find the container with id a509f13d33e744d36eddfe9b60e1580eb67a7ea53dfc0e6258e7e262ab5dab08
Nov 03 20:34:59.652294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:59.652215    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7" event={"ID":"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef","Type":"ContainerStarted","Data":"a509f13d33e744d36eddfe9b60e1580eb67a7ea53dfc0e6258e7e262ab5dab08"}
Nov 03 20:34:59.749291 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:59.749110    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:59.907491 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:34:59.907441    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:00.578030 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.577993    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"]
Nov 03 20:35:00.583289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.583261    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.632845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.632792    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:35:00.660128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.658301    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spnvt\""
Nov 03 20:35:00.681991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.681957    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"]
Nov 03 20:35:00.702200 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702166    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67311d6-9cf7-4985-abb1-1cc9763e6156-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.702391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702216    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.702391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702262    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.702391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702333    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.702588 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702440    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.702588 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702474    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtj67\" (UniqueName: \"kubernetes.io/projected/e67311d6-9cf7-4985-abb1-1cc9763e6156-kube-api-access-xtj67\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.702588 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702513    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.702588 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702540    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.702588 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.702564    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.703473 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.703331    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803583    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803642    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtj67\" (UniqueName: \"kubernetes.io/projected/e67311d6-9cf7-4985-abb1-1cc9763e6156-kube-api-access-xtj67\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803684    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803713    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803738    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803844    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67311d6-9cf7-4985-abb1-1cc9763e6156-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803874    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803920    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.803944    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.805592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.804792    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.807675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.806852    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67311d6-9cf7-4985-abb1-1cc9763e6156-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.807675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.807115    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.807675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.807342    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.807675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.807569    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.810066 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.809987    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.811579 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.811430    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.869864 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.869787    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67311d6-9cf7-4985-abb1-1cc9763e6156-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.873756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.873066    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtj67\" (UniqueName: \"kubernetes.io/projected/e67311d6-9cf7-4985-abb1-1cc9763e6156-kube-api-access-xtj67\") pod \"maas-default-gateway-openshift-default-696f47c6c-rrffz\" (UID: \"e67311d6-9cf7-4985-abb1-1cc9763e6156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.888896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.887884    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"]
Nov 03 20:35:00.894763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.894122    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:00.903974 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.903947    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:00.924492 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.924456    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:00.945353 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.945319    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"]
Nov 03 20:35:00.945765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:00.945741    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-8vjln\""
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.009963    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.010067    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84704e32-8624-49c8-b585-a12ee07f5496-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.010128    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84704e32-8624-49c8-b585-a12ee07f5496-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.010167    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.010223    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvvhm\" (UniqueName: \"kubernetes.io/projected/84704e32-8624-49c8-b585-a12ee07f5496-kube-api-access-tvvhm\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.010277    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.010366    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.010396    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84704e32-8624-49c8-b585-a12ee07f5496-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.011989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.010430    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.090560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.090277    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111153    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111205    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84704e32-8624-49c8-b585-a12ee07f5496-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111240    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111274    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111348    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84704e32-8624-49c8-b585-a12ee07f5496-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111376    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84704e32-8624-49c8-b585-a12ee07f5496-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111412    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111482    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvhm\" (UniqueName: \"kubernetes.io/projected/84704e32-8624-49c8-b585-a12ee07f5496-kube-api-access-tvvhm\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.111535    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.112073    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.112401    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.113084    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84704e32-8624-49c8-b585-a12ee07f5496-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.114444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.113390    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.115633 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.115416    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.127709 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.127593    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84704e32-8624-49c8-b585-a12ee07f5496-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.130060 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.129969    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84704e32-8624-49c8-b585-a12ee07f5496-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.173157 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.173045    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84704e32-8624-49c8-b585-a12ee07f5496-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.176316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.176258    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvhm\" (UniqueName: \"kubernetes.io/projected/84704e32-8624-49c8-b585-a12ee07f5496-kube-api-access-tvvhm\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v\" (UID: \"84704e32-8624-49c8-b585-a12ee07f5496\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.214435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.214113    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:01.225677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.225160    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"]
Nov 03 20:35:01.231275 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.231130    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.289327 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.289288    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"]
Nov 03 20:35:01.314331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.313917    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.314331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.313991    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.314331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.314188    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvpfl\" (UniqueName: \"kubernetes.io/projected/f8ce5454-71f7-4856-9c29-83737f7bbba3-kube-api-access-cvpfl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.373141 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.373018    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"]
Nov 03 20:35:01.384337 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:35:01.384262    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67311d6_9cf7_4985_abb1_1cc9763e6156.slice/crio-64f6ba74c2c948dd235f3c525391a753fddca32ffaf2998a023af501e5d6e89c WatchSource:0}: Error finding container 64f6ba74c2c948dd235f3c525391a753fddca32ffaf2998a023af501e5d6e89c: Status 404 returned error can't find the container with id 64f6ba74c2c948dd235f3c525391a753fddca32ffaf2998a023af501e5d6e89c
Nov 03 20:35:01.415183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.415048    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvpfl\" (UniqueName: \"kubernetes.io/projected/f8ce5454-71f7-4856-9c29-83737f7bbba3-kube-api-access-cvpfl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.415183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.415156    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.415417 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.415189    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.415662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.415609    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.416242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.416217    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.474648 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.474613    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvpfl\" (UniqueName: \"kubernetes.io/projected/f8ce5454-71f7-4856-9c29-83737f7bbba3-kube-api-access-cvpfl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.555372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.555334    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:01.625110 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.625052    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"]
Nov 03 20:35:01.689372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.689304    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" event={"ID":"e67311d6-9cf7-4985-abb1-1cc9763e6156","Type":"ContainerStarted","Data":"64f6ba74c2c948dd235f3c525391a753fddca32ffaf2998a023af501e5d6e89c"}
Nov 03 20:35:01.697221 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.691907    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v" event={"ID":"84704e32-8624-49c8-b585-a12ee07f5496","Type":"ContainerStarted","Data":"5a87484f76d4e5a3bf661ee6a66794f4466bf38f43a97dbbe03a7bf3a07faf3e"}
Nov 03 20:35:01.995924 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:01.995845    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"]
Nov 03 20:35:02.706130 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:02.706058    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp" event={"ID":"f8ce5454-71f7-4856-9c29-83737f7bbba3","Type":"ContainerStarted","Data":"8d742eee5e8dfed44c36f42b30990f47365e2668803b6b401360d7cdb1f1e7e8"}
Nov 03 20:35:05.265348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:05.265274    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:09.797185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:09.796702    5196 generic.go:356] "Generic (PLEG): container finished" podID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerID="01d56f7b0d039c2df869d950213b80da68da36266e26832ddfcf9b7b31bb2b53" exitCode=0
Nov 03 20:35:09.797185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:09.796890    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m" event={"ID":"a6144e77-a0f7-4020-84a3-24fd5d08733f","Type":"ContainerDied","Data":"01d56f7b0d039c2df869d950213b80da68da36266e26832ddfcf9b7b31bb2b53"}
Nov 03 20:35:09.799955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:09.799926    5196 generic.go:356] "Generic (PLEG): container finished" podID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerID="97f8dbcfb14232e4b0a71555198303d84c3ebce6ae841bbe25785c9becdf17ba" exitCode=0
Nov 03 20:35:09.800017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:09.799975    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k" event={"ID":"7f8142b6-7bcc-480f-b049-280fa3fcd205","Type":"ContainerDied","Data":"97f8dbcfb14232e4b0a71555198303d84c3ebce6ae841bbe25785c9becdf17ba"}
Nov 03 20:35:10.810330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:10.808419    5196 generic.go:356] "Generic (PLEG): container finished" podID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerID="6046cc7e7fe8ce406b19d2d9af9cdf78cea4380bdab02f21cf8b9422067fcdc0" exitCode=0
Nov 03 20:35:10.810330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:10.808557    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7" event={"ID":"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef","Type":"ContainerDied","Data":"6046cc7e7fe8ce406b19d2d9af9cdf78cea4380bdab02f21cf8b9422067fcdc0"}
Nov 03 20:35:10.814058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:10.814027    5196 generic.go:356] "Generic (PLEG): container finished" podID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerID="7e6b433e49d7814b5d4dc0e49f74cf284f24dd278f1aab0c75da7115450d4c6d" exitCode=0
Nov 03 20:35:10.814225 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:10.814120    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm" event={"ID":"53b45b22-3c6c-4c12-8800-2b8b8be5d358","Type":"ContainerDied","Data":"7e6b433e49d7814b5d4dc0e49f74cf284f24dd278f1aab0c75da7115450d4c6d"}
Nov 03 20:35:11.129407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:11.129360    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:11.129556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:11.129445    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:11.129556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:11.129486    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:11.503321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:11.503191    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:11.822191 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:11.822154    5196 generic.go:356] "Generic (PLEG): container finished" podID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerID="af5adc97b53c17e3669da8e8e8162f7ba923f7af22c12754276f2414d80eff16" exitCode=0
Nov 03 20:35:11.822596 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:11.822243    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp" event={"ID":"354e8583-212f-49bd-9bb0-118431b5d3f0","Type":"ContainerDied","Data":"af5adc97b53c17e3669da8e8e8162f7ba923f7af22c12754276f2414d80eff16"}
Nov 03 20:35:11.824313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:11.824278    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v" event={"ID":"84704e32-8624-49c8-b585-a12ee07f5496","Type":"ContainerStarted","Data":"9e5b7e241a749237a1857f5e8c95e2ca8ef9bb3e0aeb30c849f0929da42f892a"}
Nov 03 20:35:12.040803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.040707    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v" podStartSLOduration=2.563184865 podStartE2EDuration="12.040692496s" totalImagesPullingTime="9.477507631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:12.040024399 +0000 UTC m=+352.043983143" watchObservedRunningTime="2026-11-03 20:35:12.040692496 +0000 UTC m=+352.044651219"
Nov 03 20:35:12.214683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.214647    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:12.219463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.219434    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:12.830418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.830386    5196 generic.go:356] "Generic (PLEG): container finished" podID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerID="f7dc20dc8ebd8a781cf5b3c5648084a6c046fc314accc3b8ed1b0adb12d385b8" exitCode=0
Nov 03 20:35:12.830875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.830457    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k" event={"ID":"7f8142b6-7bcc-480f-b049-280fa3fcd205","Type":"ContainerDied","Data":"f7dc20dc8ebd8a781cf5b3c5648084a6c046fc314accc3b8ed1b0adb12d385b8"}
Nov 03 20:35:12.832483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.832452    5196 generic.go:356] "Generic (PLEG): container finished" podID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerID="eee26761b10e62af6fbbe0aa1ca71609e0c213a8da36f0ef8cdbf94a3d6ac0fe" exitCode=0
Nov 03 20:35:12.832553 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.832521    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7" event={"ID":"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef","Type":"ContainerDied","Data":"eee26761b10e62af6fbbe0aa1ca71609e0c213a8da36f0ef8cdbf94a3d6ac0fe"}
Nov 03 20:35:12.834254 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.834231    5196 generic.go:356] "Generic (PLEG): container finished" podID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerID="554a9000df4b7d7efefe5aedd810343fb30472ed5431dd7d7ab000e57b5658fb" exitCode=0
Nov 03 20:35:12.834345 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.834302    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs" event={"ID":"b2138c4d-ff75-49d4-8ff4-94498408efd3","Type":"ContainerDied","Data":"554a9000df4b7d7efefe5aedd810343fb30472ed5431dd7d7ab000e57b5658fb"}
Nov 03 20:35:12.836434 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.836411    5196 generic.go:356] "Generic (PLEG): container finished" podID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerID="72f944431306a0dc6649c37009676e28dc70eaf4e2c031a81823ebe6cd9dbf5d" exitCode=0
Nov 03 20:35:12.836533 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.836493    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm" event={"ID":"53b45b22-3c6c-4c12-8800-2b8b8be5d358","Type":"ContainerDied","Data":"72f944431306a0dc6649c37009676e28dc70eaf4e2c031a81823ebe6cd9dbf5d"}
Nov 03 20:35:12.838491 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.838469    5196 generic.go:356] "Generic (PLEG): container finished" podID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerID="ab1cf0cb1c52e7ab23a2b81e35908ded152a0924634f9c54bfca076856503bc3" exitCode=0
Nov 03 20:35:12.838564 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.838543    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m" event={"ID":"a6144e77-a0f7-4020-84a3-24fd5d08733f","Type":"ContainerDied","Data":"ab1cf0cb1c52e7ab23a2b81e35908ded152a0924634f9c54bfca076856503bc3"}
Nov 03 20:35:12.840176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.840153    5196 generic.go:356] "Generic (PLEG): container finished" podID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerID="add39cc78bb991104f5b96c68532de71bddb33fb48bac1057f1de443549fcfa1" exitCode=0
Nov 03 20:35:12.840279 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.840230    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp" event={"ID":"f8ce5454-71f7-4856-9c29-83737f7bbba3","Type":"ContainerDied","Data":"add39cc78bb991104f5b96c68532de71bddb33fb48bac1057f1de443549fcfa1"}
Nov 03 20:35:12.842056 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.842036    5196 generic.go:356] "Generic (PLEG): container finished" podID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerID="3fe484cee47ae73ad865fe0d1f0809239a728d0b6558ccb3b6a401469bde7a17" exitCode=0
Nov 03 20:35:12.842156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.842133    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25" event={"ID":"c94825d4-98c3-4c4b-8dd2-de5d61b6f224","Type":"ContainerDied","Data":"3fe484cee47ae73ad865fe0d1f0809239a728d0b6558ccb3b6a401469bde7a17"}
Nov 03 20:35:12.842755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.842710    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:12.845291 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.845269    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v"
Nov 03 20:35:12.926812 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.926766    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:12.926944 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.926918    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:12.927036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:12.927013    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:13.848903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:13.848865    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" event={"ID":"e67311d6-9cf7-4985-abb1-1cc9763e6156","Type":"ContainerStarted","Data":"e4c63424585f803c2fa6853f00e605df701db7da7339dd5d59aca5c83cf41371"}
Nov 03 20:35:13.904951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:13.904910    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:13.911182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:13.911151    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:13.960230 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:13.960159    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" podStartSLOduration=2.43193287 podStartE2EDuration="13.960141324s" totalImagesPullingTime="11.528208454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:13.958974485 +0000 UTC m=+353.962933208" watchObservedRunningTime="2026-11-03 20:35:13.960141324 +0000 UTC m=+353.964100048"
Nov 03 20:35:14.858364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:14.858327    5196 generic.go:356] "Generic (PLEG): container finished" podID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerID="173423280c0736140582487b610d8b1e0b1bb083564f52da247096b4d0e2b2be" exitCode=0
Nov 03 20:35:14.859741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:14.859713    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp" event={"ID":"f8ce5454-71f7-4856-9c29-83737f7bbba3","Type":"ContainerDied","Data":"173423280c0736140582487b610d8b1e0b1bb083564f52da247096b4d0e2b2be"}
Nov 03 20:35:14.861608 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:14.861473    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:14.864945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:14.864926    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:35:15.877334 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:15.877288    5196 generic.go:356] "Generic (PLEG): container finished" podID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerID="df1c1a368085599abef93e556d30e2d53326dc9efc7f43f3b59fad866e483c3c" exitCode=0
Nov 03 20:35:15.878641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:15.878611    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp" event={"ID":"354e8583-212f-49bd-9bb0-118431b5d3f0","Type":"ContainerDied","Data":"df1c1a368085599abef93e556d30e2d53326dc9efc7f43f3b59fad866e483c3c"}
Nov 03 20:35:18.513032 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:35:18.512994    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:35:19.922515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:19.921343    5196 generic.go:356] "Generic (PLEG): container finished" podID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerID="c07bd2c6ea8b2fc4657b0477d7527b8a2e64a8e117420011ff3b5c3ba777298f" exitCode=0
Nov 03 20:35:19.922515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:19.921517    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25" event={"ID":"c94825d4-98c3-4c4b-8dd2-de5d61b6f224","Type":"ContainerDied","Data":"c07bd2c6ea8b2fc4657b0477d7527b8a2e64a8e117420011ff3b5c3ba777298f"}
Nov 03 20:35:19.937980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:19.937945    5196 generic.go:356] "Generic (PLEG): container finished" podID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerID="3660153f0a073fea7008c72a6db0a734efcd4d8b6a7bb7fe4cbddb91d23c11d0" exitCode=0
Nov 03 20:35:19.938163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:19.938038    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs" event={"ID":"b2138c4d-ff75-49d4-8ff4-94498408efd3","Type":"ContainerDied","Data":"3660153f0a073fea7008c72a6db0a734efcd4d8b6a7bb7fe4cbddb91d23c11d0"}
Nov 03 20:35:29.028193 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:29.028042    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k" event={"ID":"7f8142b6-7bcc-480f-b049-280fa3fcd205","Type":"ContainerStarted","Data":"73d62c74e86ff58ec348cbfee3d176c2e2762670dbe4a4a7d7ef0a5bcab3bed2"}
Nov 03 20:35:29.138692 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:29.138626    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k" podStartSLOduration=2.949739559 podStartE2EDuration="36.138607079s" totalImagesPullingTime="33.18886752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:29.136942456 +0000 UTC m=+369.140901208" watchObservedRunningTime="2026-11-03 20:35:29.138607079 +0000 UTC m=+369.142565807"
Nov 03 20:35:30.042341 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:30.041888    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm" event={"ID":"53b45b22-3c6c-4c12-8800-2b8b8be5d358","Type":"ContainerStarted","Data":"b726bac9e81f54c697d3d8c6aaa3c9fbb3c019e8a3667d8ccee82ddf9aaa1013"}
Nov 03 20:35:30.047359 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:30.047253    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m" event={"ID":"a6144e77-a0f7-4020-84a3-24fd5d08733f","Type":"ContainerStarted","Data":"c722cde17e81255f613547e6d820a3f5fc9cf020cfc8d38030fae6d4705fb8c8"}
Nov 03 20:35:30.051315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:30.051279    5196 generic.go:356] "Generic (PLEG): container finished" podID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerID="73d62c74e86ff58ec348cbfee3d176c2e2762670dbe4a4a7d7ef0a5bcab3bed2" exitCode=0
Nov 03 20:35:30.051463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:30.051377    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k" event={"ID":"7f8142b6-7bcc-480f-b049-280fa3fcd205","Type":"ContainerDied","Data":"73d62c74e86ff58ec348cbfee3d176c2e2762670dbe4a4a7d7ef0a5bcab3bed2"}
Nov 03 20:35:30.055929 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:30.055893    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7" event={"ID":"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef","Type":"ContainerStarted","Data":"25c023528be9c8e60b71f87d5078d9b9fe745be8096d2d81d932cccc315e03d5"}
Nov 03 20:35:30.165152 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:30.165008    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm" podStartSLOduration=3.061643371 podStartE2EDuration="37.164991722s" totalImagesPullingTime="34.103348351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:30.163810672 +0000 UTC m=+370.167769417" watchObservedRunningTime="2026-11-03 20:35:30.164991722 +0000 UTC m=+370.168950437"
Nov 03 20:35:30.387376 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:30.387321    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m" podStartSLOduration=2.48876688 podStartE2EDuration="37.387300999s" totalImagesPullingTime="34.898534119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:30.385150773 +0000 UTC m=+370.389109528" watchObservedRunningTime="2026-11-03 20:35:30.387300999 +0000 UTC m=+370.391259724"
Nov 03 20:35:30.498369 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:30.498301    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7" podStartSLOduration=3.049259537 podStartE2EDuration="32.498281046s" totalImagesPullingTime="29.449021509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:30.494448297 +0000 UTC m=+370.498407061" watchObservedRunningTime="2026-11-03 20:35:30.498281046 +0000 UTC m=+370.502239772"
Nov 03 20:35:31.065447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.065077    5196 generic.go:356] "Generic (PLEG): container finished" podID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerID="b726bac9e81f54c697d3d8c6aaa3c9fbb3c019e8a3667d8ccee82ddf9aaa1013" exitCode=0
Nov 03 20:35:31.065447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.065147    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm" event={"ID":"53b45b22-3c6c-4c12-8800-2b8b8be5d358","Type":"ContainerDied","Data":"b726bac9e81f54c697d3d8c6aaa3c9fbb3c019e8a3667d8ccee82ddf9aaa1013"}
Nov 03 20:35:31.067911 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.067878    5196 generic.go:356] "Generic (PLEG): container finished" podID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerID="c722cde17e81255f613547e6d820a3f5fc9cf020cfc8d38030fae6d4705fb8c8" exitCode=0
Nov 03 20:35:31.068078 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.067956    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m" event={"ID":"a6144e77-a0f7-4020-84a3-24fd5d08733f","Type":"ContainerDied","Data":"c722cde17e81255f613547e6d820a3f5fc9cf020cfc8d38030fae6d4705fb8c8"}
Nov 03 20:35:31.070665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.070633    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25" event={"ID":"c94825d4-98c3-4c4b-8dd2-de5d61b6f224","Type":"ContainerStarted","Data":"466847478ac386c4003ab089c93ca0b08bed29ef369b6e11c5ed3c5c9a5c999c"}
Nov 03 20:35:31.073191 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.073164    5196 generic.go:356] "Generic (PLEG): container finished" podID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerID="25c023528be9c8e60b71f87d5078d9b9fe745be8096d2d81d932cccc315e03d5" exitCode=0
Nov 03 20:35:31.073315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.073240    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7" event={"ID":"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef","Type":"ContainerDied","Data":"25c023528be9c8e60b71f87d5078d9b9fe745be8096d2d81d932cccc315e03d5"}
Nov 03 20:35:31.076010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.075984    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs" event={"ID":"b2138c4d-ff75-49d4-8ff4-94498408efd3","Type":"ContainerStarted","Data":"841bed475aeb5ea3b585f68a97512687e9c88f59d2d0ebaecd3856018f63a45b"}
Nov 03 20:35:31.259224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.259145    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:35:31.355152 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.355121    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-bundle\") pod \"7f8142b6-7bcc-480f-b049-280fa3fcd205\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") "
Nov 03 20:35:31.355349 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.355163    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8142b6-7bcc-480f-b049-280fa3fcd205-kube-api-access-b4ff6\" (UniqueName: \"kubernetes.io/projected/7f8142b6-7bcc-480f-b049-280fa3fcd205-kube-api-access-b4ff6\") pod \"7f8142b6-7bcc-480f-b049-280fa3fcd205\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") "
Nov 03 20:35:31.355349 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.355275    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-util\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-util\") pod \"7f8142b6-7bcc-480f-b049-280fa3fcd205\" (UID: \"7f8142b6-7bcc-480f-b049-280fa3fcd205\") "
Nov 03 20:35:31.355902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.355685    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-bundle" pod "7f8142b6-7bcc-480f-b049-280fa3fcd205" (UID: "7f8142b6-7bcc-480f-b049-280fa3fcd205"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:31.358274 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.358246    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8142b6-7bcc-480f-b049-280fa3fcd205-kube-api-access-b4ff6" pod "7f8142b6-7bcc-480f-b049-280fa3fcd205" (UID: "7f8142b6-7bcc-480f-b049-280fa3fcd205"). InnerVolumeSpecName "kube-api-access-b4ff6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:31.361889 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.361859    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-util" pod "7f8142b6-7bcc-480f-b049-280fa3fcd205" (UID: "7f8142b6-7bcc-480f-b049-280fa3fcd205"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:31.393133 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.393064    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25" podStartSLOduration=3.846361536 podStartE2EDuration="37.393048445s" totalImagesPullingTime="33.546686909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:31.390898358 +0000 UTC m=+371.394857128" watchObservedRunningTime="2026-11-03 20:35:31.393048445 +0000 UTC m=+371.397007171"
Nov 03 20:35:31.456464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.456431    5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.456464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.456460    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ff6\" (UniqueName: \"kubernetes.io/projected/7f8142b6-7bcc-480f-b049-280fa3fcd205-kube-api-access-b4ff6\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.456464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.456471    5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f8142b6-7bcc-480f-b049-280fa3fcd205-util\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.500399 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:31.500346    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs" podStartSLOduration=2.6877136630000003 podStartE2EDuration="35.500327332s" totalImagesPullingTime="32.812613669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:31.498363328 +0000 UTC m=+371.502322073" watchObservedRunningTime="2026-11-03 20:35:31.500327332 +0000 UTC m=+371.504286059"
Nov 03 20:35:32.082024 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.081987    5196 generic.go:356] "Generic (PLEG): container finished" podID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerID="9bbed999091410a1c4bc88246da8a71cbb2023589b819cd2bc09b8403492385f" exitCode=0
Nov 03 20:35:32.082462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.082069    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp" event={"ID":"f8ce5454-71f7-4856-9c29-83737f7bbba3","Type":"ContainerDied","Data":"9bbed999091410a1c4bc88246da8a71cbb2023589b819cd2bc09b8403492385f"}
Nov 03 20:35:32.084037 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.084012    5196 generic.go:356] "Generic (PLEG): container finished" podID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerID="466847478ac386c4003ab089c93ca0b08bed29ef369b6e11c5ed3c5c9a5c999c" exitCode=0
Nov 03 20:35:32.084178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.084123    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25" event={"ID":"c94825d4-98c3-4c4b-8dd2-de5d61b6f224","Type":"ContainerDied","Data":"466847478ac386c4003ab089c93ca0b08bed29ef369b6e11c5ed3c5c9a5c999c"}
Nov 03 20:35:32.085936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.085919    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k"
Nov 03 20:35:32.086047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.085947    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfp4k" event={"ID":"7f8142b6-7bcc-480f-b049-280fa3fcd205","Type":"ContainerDied","Data":"e8c88b9fc232812bb7eb1c20617751a7a883e7575e26082f74ef8da2f4029eae"}
Nov 03 20:35:32.086047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.085980    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8c88b9fc232812bb7eb1c20617751a7a883e7575e26082f74ef8da2f4029eae"
Nov 03 20:35:32.087889 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.087868    5196 generic.go:356] "Generic (PLEG): container finished" podID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerID="841bed475aeb5ea3b585f68a97512687e9c88f59d2d0ebaecd3856018f63a45b" exitCode=0
Nov 03 20:35:32.087983 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.087932    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs" event={"ID":"b2138c4d-ff75-49d4-8ff4-94498408efd3","Type":"ContainerDied","Data":"841bed475aeb5ea3b585f68a97512687e9c88f59d2d0ebaecd3856018f63a45b"}
Nov 03 20:35:32.258587 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.258549    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:35:32.294542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.294514    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:35:32.306533 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.306503    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:35:32.365319 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365290    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b45b22-3c6c-4c12-8800-2b8b8be5d358-kube-api-access-jmcsx\" (UniqueName: \"kubernetes.io/projected/53b45b22-3c6c-4c12-8800-2b8b8be5d358-kube-api-access-jmcsx\") pod \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") "
Nov 03 20:35:32.365499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365333    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-bundle\") pod \"a6144e77-a0f7-4020-84a3-24fd5d08733f\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") "
Nov 03 20:35:32.365499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365375    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6144e77-a0f7-4020-84a3-24fd5d08733f-kube-api-access-f9vlm\" (UniqueName: \"kubernetes.io/projected/a6144e77-a0f7-4020-84a3-24fd5d08733f-kube-api-access-f9vlm\") pod \"a6144e77-a0f7-4020-84a3-24fd5d08733f\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") "
Nov 03 20:35:32.365499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365394    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-util\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-util\") pod \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") "
Nov 03 20:35:32.365499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365429    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-bundle\") pod \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") "
Nov 03 20:35:32.365499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365472    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-util\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-util\") pod \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\" (UID: \"53b45b22-3c6c-4c12-8800-2b8b8be5d358\") "
Nov 03 20:35:32.365767 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365506    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-kube-api-access-9mdx2\" (UniqueName: \"kubernetes.io/projected/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-kube-api-access-9mdx2\") pod \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") "
Nov 03 20:35:32.365767 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365568    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-util\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-util\") pod \"a6144e77-a0f7-4020-84a3-24fd5d08733f\" (UID: \"a6144e77-a0f7-4020-84a3-24fd5d08733f\") "
Nov 03 20:35:32.365767 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365597    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-bundle\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-bundle\") pod \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\" (UID: \"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef\") "
Nov 03 20:35:32.366010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.365980    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-bundle" pod "a6144e77-a0f7-4020-84a3-24fd5d08733f" (UID: "a6144e77-a0f7-4020-84a3-24fd5d08733f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:32.366648 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.366619    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-bundle" pod "53b45b22-3c6c-4c12-8800-2b8b8be5d358" (UID: "53b45b22-3c6c-4c12-8800-2b8b8be5d358"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:32.366759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.366740    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-bundle" pod "69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" (UID: "69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:32.368337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.368299    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6144e77-a0f7-4020-84a3-24fd5d08733f-kube-api-access-f9vlm" pod "a6144e77-a0f7-4020-84a3-24fd5d08733f" (UID: "a6144e77-a0f7-4020-84a3-24fd5d08733f"). InnerVolumeSpecName "kube-api-access-f9vlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:32.368714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.368693    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b45b22-3c6c-4c12-8800-2b8b8be5d358-kube-api-access-jmcsx" pod "53b45b22-3c6c-4c12-8800-2b8b8be5d358" (UID: "53b45b22-3c6c-4c12-8800-2b8b8be5d358"). InnerVolumeSpecName "kube-api-access-jmcsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:32.370004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.369975    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-kube-api-access-9mdx2" pod "69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" (UID: "69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef"). InnerVolumeSpecName "kube-api-access-9mdx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:32.373279 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.373251    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-util" pod "53b45b22-3c6c-4c12-8800-2b8b8be5d358" (UID: "53b45b22-3c6c-4c12-8800-2b8b8be5d358"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:32.374155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.374131    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-util" pod "69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" (UID: "69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:32.374898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.374877    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-util" pod "a6144e77-a0f7-4020-84a3-24fd5d08733f" (UID: "a6144e77-a0f7-4020-84a3-24fd5d08733f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.466952    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmcsx\" (UniqueName: \"kubernetes.io/projected/53b45b22-3c6c-4c12-8800-2b8b8be5d358-kube-api-access-jmcsx\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.466986    5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.466996    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9vlm\" (UniqueName: \"kubernetes.io/projected/a6144e77-a0f7-4020-84a3-24fd5d08733f-kube-api-access-f9vlm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.467005    5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-util\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.467013    5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.467021    5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b45b22-3c6c-4c12-8800-2b8b8be5d358-util\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.467029    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mdx2\" (UniqueName: \"kubernetes.io/projected/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-kube-api-access-9mdx2\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.467036    5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6144e77-a0f7-4020-84a3-24fd5d08733f-util\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:32.467041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:32.467044    5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.094964 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.094932    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7"
Nov 03 20:35:33.094964 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.094947    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l7kr7" event={"ID":"69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef","Type":"ContainerDied","Data":"a509f13d33e744d36eddfe9b60e1580eb67a7ea53dfc0e6258e7e262ab5dab08"}
Nov 03 20:35:33.095486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.094991    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a509f13d33e744d36eddfe9b60e1580eb67a7ea53dfc0e6258e7e262ab5dab08"
Nov 03 20:35:33.097268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.097239    5196 generic.go:356] "Generic (PLEG): container finished" podID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerID="6a2ae7007549b76f46cddcef5263aa9aa68baa63bf4a3788ec759fd5924d406b" exitCode=0
Nov 03 20:35:33.097389 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.097284    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp" event={"ID":"354e8583-212f-49bd-9bb0-118431b5d3f0","Type":"ContainerDied","Data":"6a2ae7007549b76f46cddcef5263aa9aa68baa63bf4a3788ec759fd5924d406b"}
Nov 03 20:35:33.099220 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.099200    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm" event={"ID":"53b45b22-3c6c-4c12-8800-2b8b8be5d358","Type":"ContainerDied","Data":"2077ca5d4b21667504e52718a319f9306d7e19cb0ed84d231678e1438f584661"}
Nov 03 20:35:33.099220 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.099215    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnpfm"
Nov 03 20:35:33.099390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.099230    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2077ca5d4b21667504e52718a319f9306d7e19cb0ed84d231678e1438f584661"
Nov 03 20:35:33.101433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.101412    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m"
Nov 03 20:35:33.101567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.101450    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj2q9m" event={"ID":"a6144e77-a0f7-4020-84a3-24fd5d08733f","Type":"ContainerDied","Data":"a13daaabbdfdb854a1fe23ecfc61e93982b75c06df2c5c88aa07603f0fbd0a9f"}
Nov 03 20:35:33.101567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.101489    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a13daaabbdfdb854a1fe23ecfc61e93982b75c06df2c5c88aa07603f0fbd0a9f"
Nov 03 20:35:33.265200 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.265158    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:35:33.268378 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.268356    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:33.294727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.294705    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:35:33.375285 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375255    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-util\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-util\") pod \"f8ce5454-71f7-4856-9c29-83737f7bbba3\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") "
Nov 03 20:35:33.375489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375308    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-util\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-util\") pod \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") "
Nov 03 20:35:33.375489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375352    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-bundle\") pod \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") "
Nov 03 20:35:33.375489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375385    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-bundle\") pod \"b2138c4d-ff75-49d4-8ff4-94498408efd3\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") "
Nov 03 20:35:33.375489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375419    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ce5454-71f7-4856-9c29-83737f7bbba3-kube-api-access-cvpfl\" (UniqueName: \"kubernetes.io/projected/f8ce5454-71f7-4856-9c29-83737f7bbba3-kube-api-access-cvpfl\") pod \"f8ce5454-71f7-4856-9c29-83737f7bbba3\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") "
Nov 03 20:35:33.375489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375456    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-util\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-util\") pod \"b2138c4d-ff75-49d4-8ff4-94498408efd3\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") "
Nov 03 20:35:33.375741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375494    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2138c4d-ff75-49d4-8ff4-94498408efd3-kube-api-access-jdx78\" (UniqueName: \"kubernetes.io/projected/b2138c4d-ff75-49d4-8ff4-94498408efd3-kube-api-access-jdx78\") pod \"b2138c4d-ff75-49d4-8ff4-94498408efd3\" (UID: \"b2138c4d-ff75-49d4-8ff4-94498408efd3\") "
Nov 03 20:35:33.375741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375533    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-kube-api-access-nm64h\" (UniqueName: \"kubernetes.io/projected/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-kube-api-access-nm64h\") pod \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\" (UID: \"c94825d4-98c3-4c4b-8dd2-de5d61b6f224\") "
Nov 03 20:35:33.375741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.375560    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-bundle\") pod \"f8ce5454-71f7-4856-9c29-83737f7bbba3\" (UID: \"f8ce5454-71f7-4856-9c29-83737f7bbba3\") "
Nov 03 20:35:33.376383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.376306    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-bundle" pod "f8ce5454-71f7-4856-9c29-83737f7bbba3" (UID: "f8ce5454-71f7-4856-9c29-83737f7bbba3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:33.376383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.376322    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-bundle" pod "c94825d4-98c3-4c4b-8dd2-de5d61b6f224" (UID: "c94825d4-98c3-4c4b-8dd2-de5d61b6f224"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:33.376547 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.376386    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-bundle" pod "b2138c4d-ff75-49d4-8ff4-94498408efd3" (UID: "b2138c4d-ff75-49d4-8ff4-94498408efd3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:33.378151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.378121    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2138c4d-ff75-49d4-8ff4-94498408efd3-kube-api-access-jdx78" pod "b2138c4d-ff75-49d4-8ff4-94498408efd3" (UID: "b2138c4d-ff75-49d4-8ff4-94498408efd3"). InnerVolumeSpecName "kube-api-access-jdx78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:33.378346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.378326    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ce5454-71f7-4856-9c29-83737f7bbba3-kube-api-access-cvpfl" pod "f8ce5454-71f7-4856-9c29-83737f7bbba3" (UID: "f8ce5454-71f7-4856-9c29-83737f7bbba3"). InnerVolumeSpecName "kube-api-access-cvpfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:33.378426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.378409    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-kube-api-access-nm64h" pod "c94825d4-98c3-4c4b-8dd2-de5d61b6f224" (UID: "c94825d4-98c3-4c4b-8dd2-de5d61b6f224"). InnerVolumeSpecName "kube-api-access-nm64h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:33.382180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.382158    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-util" pod "c94825d4-98c3-4c4b-8dd2-de5d61b6f224" (UID: "c94825d4-98c3-4c4b-8dd2-de5d61b6f224"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:33.382337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.382316    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-util" pod "f8ce5454-71f7-4856-9c29-83737f7bbba3" (UID: "f8ce5454-71f7-4856-9c29-83737f7bbba3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:33.382471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.382448    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-util" pod "b2138c4d-ff75-49d4-8ff4-94498408efd3" (UID: "b2138c4d-ff75-49d4-8ff4-94498408efd3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:33.476702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476624    5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.476702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476664    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvpfl\" (UniqueName: \"kubernetes.io/projected/f8ce5454-71f7-4856-9c29-83737f7bbba3-kube-api-access-cvpfl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.476702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476680    5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2138c4d-ff75-49d4-8ff4-94498408efd3-util\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.476702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476695    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdx78\" (UniqueName: \"kubernetes.io/projected/b2138c4d-ff75-49d4-8ff4-94498408efd3-kube-api-access-jdx78\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.476927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476709    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm64h\" (UniqueName: \"kubernetes.io/projected/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-kube-api-access-nm64h\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.476927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476722    5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.476927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476735    5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8ce5454-71f7-4856-9c29-83737f7bbba3-util\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.476927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476748    5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-util\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.476927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.476760    5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c94825d4-98c3-4c4b-8dd2-de5d61b6f224-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.525421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:33.525374    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:34.109112 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.109068    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25"
Nov 03 20:35:34.109112 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.109075    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448vv25" event={"ID":"c94825d4-98c3-4c4b-8dd2-de5d61b6f224","Type":"ContainerDied","Data":"5882ed4bdf50b25aaf19fdbe0508ea16827ca37073b965edf1a7a458f50320e3"}
Nov 03 20:35:34.109767 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.109146    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5882ed4bdf50b25aaf19fdbe0508ea16827ca37073b965edf1a7a458f50320e3"
Nov 03 20:35:34.119765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.119737    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs"
Nov 03 20:35:34.119991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.119704    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5hqqs" event={"ID":"b2138c4d-ff75-49d4-8ff4-94498408efd3","Type":"ContainerDied","Data":"cddae8953e59c36afbb817dc87c7cfb151302f336207b4b00a970952f794df08"}
Nov 03 20:35:34.120081 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.120057    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cddae8953e59c36afbb817dc87c7cfb151302f336207b4b00a970952f794df08"
Nov 03 20:35:34.122181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.122149    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp" event={"ID":"f8ce5454-71f7-4856-9c29-83737f7bbba3","Type":"ContainerDied","Data":"8d742eee5e8dfed44c36f42b30990f47365e2668803b6b401360d7cdb1f1e7e8"}
Nov 03 20:35:34.122316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.122224    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d742eee5e8dfed44c36f42b30990f47365e2668803b6b401360d7cdb1f1e7e8"
Nov 03 20:35:34.122316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.122280    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gdhvp"
Nov 03 20:35:34.259686 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.259661    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:35:34.383443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.383409    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-util\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-util\") pod \"354e8583-212f-49bd-9bb0-118431b5d3f0\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") "
Nov 03 20:35:34.383602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.383459    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354e8583-212f-49bd-9bb0-118431b5d3f0-kube-api-access-5dwmg\" (UniqueName: \"kubernetes.io/projected/354e8583-212f-49bd-9bb0-118431b5d3f0-kube-api-access-5dwmg\") pod \"354e8583-212f-49bd-9bb0-118431b5d3f0\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") "
Nov 03 20:35:34.383602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.383501    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-bundle\") pod \"354e8583-212f-49bd-9bb0-118431b5d3f0\" (UID: \"354e8583-212f-49bd-9bb0-118431b5d3f0\") "
Nov 03 20:35:34.383996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.383967    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-bundle" pod "354e8583-212f-49bd-9bb0-118431b5d3f0" (UID: "354e8583-212f-49bd-9bb0-118431b5d3f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:34.385759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.385726    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354e8583-212f-49bd-9bb0-118431b5d3f0-kube-api-access-5dwmg" pod "354e8583-212f-49bd-9bb0-118431b5d3f0" (UID: "354e8583-212f-49bd-9bb0-118431b5d3f0"). InnerVolumeSpecName "kube-api-access-5dwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:34.388247 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.388226    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-util" pod "354e8583-212f-49bd-9bb0-118431b5d3f0" (UID: "354e8583-212f-49bd-9bb0-118431b5d3f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:34.484281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.484247    5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-util\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:34.484281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.484274    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dwmg\" (UniqueName: \"kubernetes.io/projected/354e8583-212f-49bd-9bb0-118431b5d3f0-kube-api-access-5dwmg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:34.484281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:34.484284    5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/354e8583-212f-49bd-9bb0-118431b5d3f0-bundle\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:35.129087 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:35.129059    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp"
Nov 03 20:35:35.129087 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:35.129072    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qbpbp" event={"ID":"354e8583-212f-49bd-9bb0-118431b5d3f0","Type":"ContainerDied","Data":"3335652e685feb38570f257c918c0c9f9de627754c821436133eac8fe5063853"}
Nov 03 20:35:35.129601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:35.129139    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3335652e685feb38570f257c918c0c9f9de627754c821436133eac8fe5063853"
Nov 03 20:35:35.380716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:35.380608    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:35.555047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:35.555009    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.062776 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.062741    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.239888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.239849    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.274061 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274021    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"]
Nov 03 20:35:36.274459 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274440    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerName="pull"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274462    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerName="pull"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274475    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerName="extract"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274483    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerName="extract"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274499    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerName="pull"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274507    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerName="pull"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274522    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerName="pull"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274529    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerName="pull"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274541    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerName="extract"
Nov 03 20:35:36.274545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274548    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274562    5196 cpu_manager.go:414] "containerMap: removing container" podUID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274569    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274583    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274591    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274603    5196 cpu_manager.go:414] "containerMap: removing container" podUID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274609    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274618    5196 cpu_manager.go:414] "containerMap: removing container" podUID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274626    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274637    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274644    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274656    5196 cpu_manager.go:414] "containerMap: removing container" podUID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274663    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274678    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274685    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274694    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274700    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274712    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274719    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274729    5196 cpu_manager.go:414] "containerMap: removing container" podUID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274736    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerName="pull"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274750    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274757    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274769    5196 cpu_manager.go:414] "containerMap: removing container" podUID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274777    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274796    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274803    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274815    5196 cpu_manager.go:414] "containerMap: removing container" podUID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274821    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274830    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274837    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274846    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274853    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274862    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274869    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerName="util"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274878    5196 cpu_manager.go:414] "containerMap: removing container" podUID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274885    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274897    5196 cpu_manager.go:414] "containerMap: removing container" podUID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274903    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.274986    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="354e8583-212f-49bd-9bb0-118431b5d3f0" containerName="extract"
Nov 03 20:35:36.274987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.275002    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8142b6-7bcc-480f-b049-280fa3fcd205" containerName="extract"
Nov 03 20:35:36.276659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.275013    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94825d4-98c3-4c4b-8dd2-de5d61b6f224" containerName="extract"
Nov 03 20:35:36.276659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.275024    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6144e77-a0f7-4020-84a3-24fd5d08733f" containerName="extract"
Nov 03 20:35:36.276659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.275033    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b45b22-3c6c-4c12-8800-2b8b8be5d358" containerName="extract"
Nov 03 20:35:36.276659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.275042    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d0fa35-02fe-48a0-a2b2-84c48fe7a2ef" containerName="extract"
Nov 03 20:35:36.276659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.275051    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ce5454-71f7-4856-9c29-83737f7bbba3" containerName="extract"
Nov 03 20:35:36.276659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.275060    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2138c4d-ff75-49d4-8ff4-94498408efd3" containerName="extract"
Nov 03 20:35:36.277813 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.277793    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.330520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.330483    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 20:35:36.330687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.330501    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 20:35:36.330687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.330490    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 20:35:36.377041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.377007    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 20:35:36.377041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.377013    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 20:35:36.377041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.377039    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-s65qj\""
Nov 03 20:35:36.392972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.392944    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"]
Nov 03 20:35:36.402567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.402532    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.402735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.402587    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.402735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.402681    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.402823 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.402809    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkltd\" (UniqueName: \"kubernetes.io/projected/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-kube-api-access-wkltd\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.402878 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.402865    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.402921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.402891    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.504118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.504066    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkltd\" (UniqueName: \"kubernetes.io/projected/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-kube-api-access-wkltd\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.504461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.504146    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.504461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.504168    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.504461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.504192    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.504461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.504210    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.504461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.504248    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.504758 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.504688    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-manifests\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.505737 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.505700    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.506787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.506761    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-tmp\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.507153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.507130    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-metrics-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.507373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.507353    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-webhook-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.507441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.507371    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-apiservice-cert\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.558431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.558396    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkltd\" (UniqueName: \"kubernetes.io/projected/b2725201-b0e4-4bc5-b3d3-e416e3e2f767-kube-api-access-wkltd\") pod \"opendatahub-operator-controller-manager-768646756f-f86vn\" (UID: \"b2725201-b0e4-4bc5-b3d3-e416e3e2f767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.590819 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.590735    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:36.674612 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.674581    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.844958 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:36.844866    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:37.076220 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:37.076183    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"]
Nov 03 20:35:37.082996 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:35:37.082967    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2725201_b0e4_4bc5_b3d3_e416e3e2f767.slice/crio-db241b8713b24f54d5d9935684f453edaec45555ea51d63e5bd089494fd0f6a8 WatchSource:0}: Error finding container db241b8713b24f54d5d9935684f453edaec45555ea51d63e5bd089494fd0f6a8: Status 404 returned error can't find the container with id db241b8713b24f54d5d9935684f453edaec45555ea51d63e5bd089494fd0f6a8
Nov 03 20:35:37.140357 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:37.140266    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn" event={"ID":"b2725201-b0e4-4bc5-b3d3-e416e3e2f767","Type":"ContainerStarted","Data":"db241b8713b24f54d5d9935684f453edaec45555ea51d63e5bd089494fd0f6a8"}
Nov 03 20:35:37.848570 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:37.848527    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:39.009176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:39.009139    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:39.217395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:39.217335    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:39.604471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:39.604414    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:39.759618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:39.759584    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:41.164863 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:41.164825    5196 generic.go:356] "Generic (PLEG): container finished" podID="b2725201-b0e4-4bc5-b3d3-e416e3e2f767" containerID="1dc23720326dd3ff4d0481e7309aa8f35e762d705580ff8b9f2fa8dcd86d0d1a" exitCode=0
Nov 03 20:35:41.165248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:41.164903    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn" event={"ID":"b2725201-b0e4-4bc5-b3d3-e416e3e2f767","Type":"ContainerDied","Data":"1dc23720326dd3ff4d0481e7309aa8f35e762d705580ff8b9f2fa8dcd86d0d1a"}
Nov 03 20:35:42.172415 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:42.172381    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn" event={"ID":"b2725201-b0e4-4bc5-b3d3-e416e3e2f767","Type":"ContainerStarted","Data":"15458f8b9fdadd8f3d00b7d88c50e3f4fafbd0449803c5cab4d74b0cdc00a597"}
Nov 03 20:35:42.172801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:42.172440    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:42.393462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:42.393342    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn" podStartSLOduration=1.47599979 podStartE2EDuration="6.393325082s" totalImagesPullingTime="4.442133638s" totalInitContainerRuntime="475.191654ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:42.345389016 +0000 UTC m=+382.349347733" watchObservedRunningTime="2026-11-03 20:35:42.393325082 +0000 UTC m=+382.397283847"
Nov 03 20:35:44.195852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:44.195801    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:44.398419 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:44.398379    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:48.176497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:48.176462    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:48.483239 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:35:48.483153    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:35:48.587714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:48.587678    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:49.379761 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:49.379713    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:50.608945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:50.608892    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:53.179639 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.179602    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn"
Nov 03 20:35:53.327574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.327544    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"]
Nov 03 20:35:53.406258 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.406233    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"
Nov 03 20:35:53.449359 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.449283    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"]
Nov 03 20:35:53.452138 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.452076    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8k2k\" (UniqueName: \"kubernetes.io/projected/ef662800-8b36-4e42-94ed-e74b73d82343-kube-api-access-p8k2k\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg\" (UID: \"ef662800-8b36-4e42-94ed-e74b73d82343\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"
Nov 03 20:35:53.452262 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.452210    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef662800-8b36-4e42-94ed-e74b73d82343-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg\" (UID: \"ef662800-8b36-4e42-94ed-e74b73d82343\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"
Nov 03 20:35:53.455882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.455830    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 20:35:53.455882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.455865    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:35:53.456063 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.456003    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-67m6l\""
Nov 03 20:35:53.553187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.553156    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef662800-8b36-4e42-94ed-e74b73d82343-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg\" (UID: \"ef662800-8b36-4e42-94ed-e74b73d82343\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"
Nov 03 20:35:53.553348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.553259    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8k2k\" (UniqueName: \"kubernetes.io/projected/ef662800-8b36-4e42-94ed-e74b73d82343-kube-api-access-p8k2k\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg\" (UID: \"ef662800-8b36-4e42-94ed-e74b73d82343\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"
Nov 03 20:35:53.553573 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.553550    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef662800-8b36-4e42-94ed-e74b73d82343-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg\" (UID: \"ef662800-8b36-4e42-94ed-e74b73d82343\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"
Nov 03 20:35:53.617911 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.617874    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8k2k\" (UniqueName: \"kubernetes.io/projected/ef662800-8b36-4e42-94ed-e74b73d82343-kube-api-access-p8k2k\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg\" (UID: \"ef662800-8b36-4e42-94ed-e74b73d82343\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"
Nov 03 20:35:53.733603 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:53.733508    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"
Nov 03 20:35:54.175657 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:54.175629    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg"]
Nov 03 20:35:54.182308 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:35:54.182281    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef662800_8b36_4e42_94ed_e74b73d82343.slice/crio-179cf9ee33af140f198bbdcd98c86b4dd61ff6434528d8e730e68c0771bd2d32 WatchSource:0}: Error finding container 179cf9ee33af140f198bbdcd98c86b4dd61ff6434528d8e730e68c0771bd2d32: Status 404 returned error can't find the container with id 179cf9ee33af140f198bbdcd98c86b4dd61ff6434528d8e730e68c0771bd2d32
Nov 03 20:35:54.229492 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:54.229458    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg" event={"ID":"ef662800-8b36-4e42-94ed-e74b73d82343","Type":"ContainerStarted","Data":"179cf9ee33af140f198bbdcd98c86b4dd61ff6434528d8e730e68c0771bd2d32"}
Nov 03 20:35:56.712105 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.712052    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"]
Nov 03 20:35:56.715876 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.715849    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:35:56.762748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.762714    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"]
Nov 03 20:35:56.778760 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.778717    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vmfgm\" (UID: \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:35:56.778958 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.778786    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w4dw\" (UniqueName: \"kubernetes.io/projected/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-kube-api-access-9w4dw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vmfgm\" (UID: \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:35:56.807004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.806971    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 20:35:56.807004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.806996    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jjnk7\""
Nov 03 20:35:56.807004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.806971    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 20:35:56.879667 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.879625    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vmfgm\" (UID: \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:35:56.879842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.879685    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w4dw\" (UniqueName: \"kubernetes.io/projected/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-kube-api-access-9w4dw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vmfgm\" (UID: \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:35:56.880535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.880501    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vmfgm\" (UID: \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:35:56.940338 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:56.940298    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w4dw\" (UniqueName: \"kubernetes.io/projected/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-kube-api-access-9w4dw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vmfgm\" (UID: \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:35:57.032181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:57.032072    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:35:57.248435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:57.248398    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg" event={"ID":"ef662800-8b36-4e42-94ed-e74b73d82343","Type":"ContainerStarted","Data":"1b0118f4b00ccf638924d387962e861977ddaff4cd2db2d2d19e4daae572ac9c"}
Nov 03 20:35:57.360085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:57.360017    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lj5gg" podStartSLOduration=1.421763971 podStartE2EDuration="4.360002585s" totalImagesPullingTime="2.938238614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:57.357946612 +0000 UTC m=+397.361905357" watchObservedRunningTime="2026-11-03 20:35:57.360002585 +0000 UTC m=+397.363961309"
Nov 03 20:35:57.514592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:57.514559    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"]
Nov 03 20:35:57.521809 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:35:57.521768    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dcfae1_34f9_4d34_b540_bc1dc9fa1254.slice/crio-1961c5156d90707cafb55f5d476b3bebc14dfcc64e79135c0c912f65ad1d42d3 WatchSource:0}: Error finding container 1961c5156d90707cafb55f5d476b3bebc14dfcc64e79135c0c912f65ad1d42d3: Status 404 returned error can't find the container with id 1961c5156d90707cafb55f5d476b3bebc14dfcc64e79135c0c912f65ad1d42d3
Nov 03 20:35:57.605895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:57.605862    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:58.254218 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:58.254183    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm" event={"ID":"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254","Type":"ContainerStarted","Data":"1961c5156d90707cafb55f5d476b3bebc14dfcc64e79135c0c912f65ad1d42d3"}
Nov 03 20:35:58.779379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:58.779336    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:59.015231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.015187    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:59.595722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.595640    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"]
Nov 03 20:35:59.617602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.617558    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"
Nov 03 20:35:59.666912 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.666876    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"]
Nov 03 20:35:59.700218 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.700173    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 20:35:59.702598 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.702560    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5c72\" (UniqueName: \"kubernetes.io/projected/c0a56f9e-a976-4628-8573-2a10b958deea-kube-api-access-j5c72\") pod \"dns-operator-controller-manager-8557cbdb84-vqvkc\" (UID: \"c0a56f9e-a976-4628-8573-2a10b958deea\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"
Nov 03 20:35:59.727466 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.727430    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4drx2\""
Nov 03 20:35:59.803591 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.803558    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5c72\" (UniqueName: \"kubernetes.io/projected/c0a56f9e-a976-4628-8573-2a10b958deea-kube-api-access-j5c72\") pod \"dns-operator-controller-manager-8557cbdb84-vqvkc\" (UID: \"c0a56f9e-a976-4628-8573-2a10b958deea\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"
Nov 03 20:35:59.859575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.859487    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5c72\" (UniqueName: \"kubernetes.io/projected/c0a56f9e-a976-4628-8573-2a10b958deea-kube-api-access-j5c72\") pod \"dns-operator-controller-manager-8557cbdb84-vqvkc\" (UID: \"c0a56f9e-a976-4628-8573-2a10b958deea\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"
Nov 03 20:35:59.950390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:35:59.950349    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"
Nov 03 20:36:00.356433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:00.356391    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"]
Nov 03 20:36:00.363359 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:36:00.363321    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a56f9e_a976_4628_8573_2a10b958deea.slice/crio-c964f7cbc58d4e7e4486bdf95fb4445fb1c1d69e95bfd4bc3a336a9569ff4d56 WatchSource:0}: Error finding container c964f7cbc58d4e7e4486bdf95fb4445fb1c1d69e95bfd4bc3a336a9569ff4d56: Status 404 returned error can't find the container with id c964f7cbc58d4e7e4486bdf95fb4445fb1c1d69e95bfd4bc3a336a9569ff4d56
Nov 03 20:36:00.983467 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:00.983424    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:01.149348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:01.149309    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:01.272882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:01.272784    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc" event={"ID":"c0a56f9e-a976-4628-8573-2a10b958deea","Type":"ContainerStarted","Data":"c964f7cbc58d4e7e4486bdf95fb4445fb1c1d69e95bfd4bc3a336a9569ff4d56"}
Nov 03 20:36:01.633771 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:01.633713    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:03.286654 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.286565    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc" event={"ID":"c0a56f9e-a976-4628-8573-2a10b958deea","Type":"ContainerStarted","Data":"fa6e480b7ff861a24052f49dcd9165dc72594dacb22f31802bdc6c3d04a0d8cf"}
Nov 03 20:36:03.286654 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.286632    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"
Nov 03 20:36:03.288331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.288302    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm" event={"ID":"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254","Type":"ContainerStarted","Data":"a341e8cb0f2d165d7b8c0e9a101475ad7404bf30e71f08af1f5a5ec4a255e425"}
Nov 03 20:36:03.288453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.288356    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:36:03.400396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.400342    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc" podStartSLOduration=1.874241667 podStartE2EDuration="4.400328185s" totalImagesPullingTime="2.526086518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:03.397638735 +0000 UTC m=+403.401597461" watchObservedRunningTime="2026-11-03 20:36:03.400328185 +0000 UTC m=+403.404286912"
Nov 03 20:36:03.512662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.512599    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm" podStartSLOduration=2.463246952 podStartE2EDuration="7.512585042s" totalImagesPullingTime="5.04933809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:03.510672054 +0000 UTC m=+403.514630807" watchObservedRunningTime="2026-11-03 20:36:03.512585042 +0000 UTC m=+403.516543772"
Nov 03 20:36:03.747811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.747775    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"]
Nov 03 20:36:03.750477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.750458    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"
Nov 03 20:36:03.800172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.800141    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pq2t8\""
Nov 03 20:36:03.805895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.805863    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"]
Nov 03 20:36:03.841824 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.841786    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzvw\" (UniqueName: \"kubernetes.io/projected/7ae07717-de88-4d8f-b9f3-42b58dbf9bba-kube-api-access-6gzvw\") pod \"limitador-operator-controller-manager-5778bb4449-cntd4\" (UID: \"7ae07717-de88-4d8f-b9f3-42b58dbf9bba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"
Nov 03 20:36:03.943110 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:03.943057    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzvw\" (UniqueName: \"kubernetes.io/projected/7ae07717-de88-4d8f-b9f3-42b58dbf9bba-kube-api-access-6gzvw\") pod \"limitador-operator-controller-manager-5778bb4449-cntd4\" (UID: \"7ae07717-de88-4d8f-b9f3-42b58dbf9bba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"
Nov 03 20:36:04.010980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:04.010907    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzvw\" (UniqueName: \"kubernetes.io/projected/7ae07717-de88-4d8f-b9f3-42b58dbf9bba-kube-api-access-6gzvw\") pod \"limitador-operator-controller-manager-5778bb4449-cntd4\" (UID: \"7ae07717-de88-4d8f-b9f3-42b58dbf9bba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"
Nov 03 20:36:04.062317 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:04.062279    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"
Nov 03 20:36:04.381105 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:04.381053    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:04.487927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:04.487896    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"]
Nov 03 20:36:04.494820 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:36:04.494780    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae07717_de88_4d8f_b9f3_42b58dbf9bba.slice/crio-51060513cf9d80a8d84e07179d4dade6829a713726db30bc0c33e2c82e0b84af WatchSource:0}: Error finding container 51060513cf9d80a8d84e07179d4dade6829a713726db30bc0c33e2c82e0b84af: Status 404 returned error can't find the container with id 51060513cf9d80a8d84e07179d4dade6829a713726db30bc0c33e2c82e0b84af
Nov 03 20:36:05.199009 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:05.198978    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:05.301746 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:05.301710    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4" event={"ID":"7ae07717-de88-4d8f-b9f3-42b58dbf9bba","Type":"ContainerStarted","Data":"51060513cf9d80a8d84e07179d4dade6829a713726db30bc0c33e2c82e0b84af"}
Nov 03 20:36:07.314497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.314462    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4" event={"ID":"7ae07717-de88-4d8f-b9f3-42b58dbf9bba","Type":"ContainerStarted","Data":"bdb627d46391f0ac680a369b01d74c0ad65251b01f59cd9617ec34567cb1cf80"}
Nov 03 20:36:07.314896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.314556    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"
Nov 03 20:36:07.424780 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.424730    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4" podStartSLOduration=2.222509178 podStartE2EDuration="4.424715554s" totalImagesPullingTime="2.202206376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:07.424132233 +0000 UTC m=+407.428090977" watchObservedRunningTime="2026-11-03 20:36:07.424715554 +0000 UTC m=+407.428674278"
Nov 03 20:36:07.742503 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.742470    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"]
Nov 03 20:36:07.745019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.744996    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"
Nov 03 20:36:07.778430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.778392    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hg45\" (UniqueName: \"kubernetes.io/projected/b4244452-ce11-46a6-8909-ecbdab87137d-kube-api-access-2hg45\") pod \"authorino-operator-7b7bb4bc77-zxpzr\" (UID: \"b4244452-ce11-46a6-8909-ecbdab87137d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"
Nov 03 20:36:07.794050 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.794027    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mlxmd\""
Nov 03 20:36:07.842756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.842725    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"]
Nov 03 20:36:07.879483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.879450    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hg45\" (UniqueName: \"kubernetes.io/projected/b4244452-ce11-46a6-8909-ecbdab87137d-kube-api-access-2hg45\") pod \"authorino-operator-7b7bb4bc77-zxpzr\" (UID: \"b4244452-ce11-46a6-8909-ecbdab87137d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"
Nov 03 20:36:07.917640 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.917605    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:07.938910 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:07.938875    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hg45\" (UniqueName: \"kubernetes.io/projected/b4244452-ce11-46a6-8909-ecbdab87137d-kube-api-access-2hg45\") pod \"authorino-operator-7b7bb4bc77-zxpzr\" (UID: \"b4244452-ce11-46a6-8909-ecbdab87137d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"
Nov 03 20:36:08.057826 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:08.057736    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"
Nov 03 20:36:08.093293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:08.093249    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:08.294736 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:08.294700    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:08.427804 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:08.427766    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"]
Nov 03 20:36:08.434067 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:36:08.434035    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4244452_ce11_46a6_8909_ecbdab87137d.slice/crio-81c616285a8709311a1437be222c8b5bcbdf37e09a42d5ac8434ea778c7d5c5c WatchSource:0}: Error finding container 81c616285a8709311a1437be222c8b5bcbdf37e09a42d5ac8434ea778c7d5c5c: Status 404 returned error can't find the container with id 81c616285a8709311a1437be222c8b5bcbdf37e09a42d5ac8434ea778c7d5c5c
Nov 03 20:36:08.468206 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:08.468165    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:08.621271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:08.621235    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:09.325006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:09.324968    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr" event={"ID":"b4244452-ce11-46a6-8909-ecbdab87137d","Type":"ContainerStarted","Data":"81c616285a8709311a1437be222c8b5bcbdf37e09a42d5ac8434ea778c7d5c5c"}
Nov 03 20:36:11.338143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:11.338070    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr" event={"ID":"b4244452-ce11-46a6-8909-ecbdab87137d","Type":"ContainerStarted","Data":"8dde65ed023f5a6b37df196601e47748122cce07ee1562c46b432be61e4a41b3"}
Nov 03 20:36:11.338607 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:11.338258    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"
Nov 03 20:36:11.448533 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:11.448432    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr" podStartSLOduration=1.773318598 podStartE2EDuration="4.448416867s" totalImagesPullingTime="2.675098269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:11.447135335 +0000 UTC m=+411.451094081" watchObservedRunningTime="2026-11-03 20:36:11.448416867 +0000 UTC m=+411.452375591"
Nov 03 20:36:13.513373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:13.513335    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:13.719312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:13.719277    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:13.917661 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:13.917624    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:14.136907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:14.136871    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:14.297390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:14.297297    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:36:14.297554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:14.297458    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vqvkc"
Nov 03 20:36:15.376586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:15.376549    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:15.985471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:15.985435    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.203752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.203717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.378809 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.378773    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.540586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.540540    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.647630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.647550    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"]
Nov 03 20:36:16.654769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.654743    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:16.709852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.709821    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"]
Nov 03 20:36:16.710900 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.710869    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.754950 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.754916    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 20:36:16.754950 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.754952    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n792z\""
Nov 03 20:36:16.756946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.756927    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"]
Nov 03 20:36:16.757302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.757280    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04366c39-0027-4379-92be-115a0979a562-config-file\") pod \"limitador-limitador-74dfcfb894-ttkgt\" (UID: \"04366c39-0027-4379-92be-115a0979a562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:16.757415 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.757400    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f49jh\" (UniqueName: \"kubernetes.io/projected/04366c39-0027-4379-92be-115a0979a562-kube-api-access-f49jh\") pod \"limitador-limitador-74dfcfb894-ttkgt\" (UID: \"04366c39-0027-4379-92be-115a0979a562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:16.759622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.759607    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.811682 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.811648    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"]
Nov 03 20:36:16.812634 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.812605    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k6sv6\""
Nov 03 20:36:16.857874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.857836    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-data\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.857874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.857887    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f49jh\" (UniqueName: \"kubernetes.io/projected/04366c39-0027-4379-92be-115a0979a562-kube-api-access-f49jh\") pod \"limitador-limitador-74dfcfb894-ttkgt\" (UID: \"04366c39-0027-4379-92be-115a0979a562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:16.858205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.857932    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.858205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.857956    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b04a8da-2555-4dfd-b88b-f86a86df7114-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.858205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.858011    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04366c39-0027-4379-92be-115a0979a562-config-file\") pod \"limitador-limitador-74dfcfb894-ttkgt\" (UID: \"04366c39-0027-4379-92be-115a0979a562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:16.858205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.858036    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-token\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.858205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.858063    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.858205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.858115    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmpz\" (UniqueName: \"kubernetes.io/projected/7b04a8da-2555-4dfd-b88b-f86a86df7114-kube-api-access-vmmpz\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.858516 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.858226    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.859010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.858968    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.859010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.858986    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04366c39-0027-4379-92be-115a0979a562-config-file\") pod \"limitador-limitador-74dfcfb894-ttkgt\" (UID: \"04366c39-0027-4379-92be-115a0979a562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:16.859205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.859026    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.902697 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.902616    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"]
Nov 03 20:36:16.913852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.913825    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f49jh\" (UniqueName: \"kubernetes.io/projected/04366c39-0027-4379-92be-115a0979a562-kube-api-access-f49jh\") pod \"limitador-limitador-74dfcfb894-ttkgt\" (UID: \"04366c39-0027-4379-92be-115a0979a562\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:16.927752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.927724    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.959994 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.959956    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.960313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.960289    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b04a8da-2555-4dfd-b88b-f86a86df7114-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.960495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.960471    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-token\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.960629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.960509    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.960629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.960544    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmpz\" (UniqueName: \"kubernetes.io/projected/7b04a8da-2555-4dfd-b88b-f86a86df7114-kube-api-access-vmmpz\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.960765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.960607    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.960765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.960676    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.960765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.960753    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.960929 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.960779    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-data\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.961656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.961215    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.961656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.961292    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b04a8da-2555-4dfd-b88b-f86a86df7114-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.961656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.961358    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.961656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.961609    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-data\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.961656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.961626    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.963270 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.963245    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.963405 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.963271    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:16.967359 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:16.967332    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:17.015560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.015524    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b04a8da-2555-4dfd-b88b-f86a86df7114-istio-token\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:17.015837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.015813    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmpz\" (UniqueName: \"kubernetes.io/projected/7b04a8da-2555-4dfd-b88b-f86a86df7114-kube-api-access-vmmpz\") pod \"data-science-gateway-data-science-gateway-class-55fd485789zcp6f\" (UID: \"7b04a8da-2555-4dfd-b88b-f86a86df7114\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:17.073059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.073019    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:17.095455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.095414    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.302521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.302418    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.347973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.347924    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"]
Nov 03 20:36:17.360542 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:36:17.360495    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04366c39_0027_4379_92be_115a0979a562.slice/crio-af67a900f09bc763ba12b64df004da764bba6384b8de7af106cd29a83433199b WatchSource:0}: Error finding container af67a900f09bc763ba12b64df004da764bba6384b8de7af106cd29a83433199b: Status 404 returned error can't find the container with id af67a900f09bc763ba12b64df004da764bba6384b8de7af106cd29a83433199b
Nov 03 20:36:17.372442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.372406    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt" event={"ID":"04366c39-0027-4379-92be-115a0979a562","Type":"ContainerStarted","Data":"af67a900f09bc763ba12b64df004da764bba6384b8de7af106cd29a83433199b"}
Nov 03 20:36:17.463342 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.463310    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"]
Nov 03 20:36:17.470061 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:36:17.470033    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b04a8da_2555_4dfd_b88b_f86a86df7114.slice/crio-73c20436a96f22b924e349b4eff9819f9ef5a3586ebb0c3e63cf7be102b7b035 WatchSource:0}: Error finding container 73c20436a96f22b924e349b4eff9819f9ef5a3586ebb0c3e63cf7be102b7b035: Status 404 returned error can't find the container with id 73c20436a96f22b924e349b4eff9819f9ef5a3586ebb0c3e63cf7be102b7b035
Nov 03 20:36:17.472835 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.472799    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:17.472919 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.472876    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:17.472919 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:17.472905    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:18.321019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:18.320989    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cntd4"
Nov 03 20:36:18.379397 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:18.379359    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f" event={"ID":"7b04a8da-2555-4dfd-b88b-f86a86df7114","Type":"ContainerStarted","Data":"60826cf58d164684177ebe3661452dfb3192ae09ca75787dc6b1ed8067ebbd6f"}
Nov 03 20:36:18.379551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:18.379418    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f" event={"ID":"7b04a8da-2555-4dfd-b88b-f86a86df7114","Type":"ContainerStarted","Data":"73c20436a96f22b924e349b4eff9819f9ef5a3586ebb0c3e63cf7be102b7b035"}
Nov 03 20:36:18.489205 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:36:18.489172    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:36:18.546509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:18.546454    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f" podStartSLOduration=2.546437494 podStartE2EDuration="2.546437494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:18.544271653 +0000 UTC m=+418.548230403" watchObservedRunningTime="2026-11-03 20:36:18.546437494 +0000 UTC m=+418.550396218"
Nov 03 20:36:19.073445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:19.073410    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:19.078156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:19.078132    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:19.385980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:19.385901    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:19.387047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:19.387023    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789zcp6f"
Nov 03 20:36:20.392149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:20.392100    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt" event={"ID":"04366c39-0027-4379-92be-115a0979a562","Type":"ContainerStarted","Data":"2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9"}
Nov 03 20:36:20.392577 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:20.392285    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:20.504301 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:20.504220    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt" podStartSLOduration=2.161743494 podStartE2EDuration="4.504200929s" totalImagesPullingTime="2.342457435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:20.499504075 +0000 UTC m=+420.503462799" watchObservedRunningTime="2026-11-03 20:36:20.504200929 +0000 UTC m=+420.508159653"
Nov 03 20:36:22.345718 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:22.345691    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zxpzr"
Nov 03 20:36:23.227873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:23.227835    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:23.389332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:23.389293    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:23.539725 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:23.539618    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:31.399312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:31.399279    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:36:33.311035 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:33.310994    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:33.465283 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:33.465243    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:36.722347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:36.722299    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"]
Nov 03 20:36:36.722809 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:36.722553    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm" podUID="d5dcfae1-34f9-4d34-b540-bc1dc9fa1254" containerName="manager" containerID="cri-o://a341e8cb0f2d165d7b8c0e9a101475ad7404bf30e71f08af1f5a5ec4a255e425" gracePeriod=10
Nov 03 20:36:37.484082 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.484048    5196 generic.go:356] "Generic (PLEG): container finished" podID="d5dcfae1-34f9-4d34-b540-bc1dc9fa1254" containerID="a341e8cb0f2d165d7b8c0e9a101475ad7404bf30e71f08af1f5a5ec4a255e425" exitCode=0
Nov 03 20:36:37.484272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.484127    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm" event={"ID":"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254","Type":"ContainerDied","Data":"a341e8cb0f2d165d7b8c0e9a101475ad7404bf30e71f08af1f5a5ec4a255e425"}
Nov 03 20:36:37.580352 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.580317    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:36:37.651946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.651867    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-kube-api-access-9w4dw\" (UniqueName: \"kubernetes.io/projected/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-kube-api-access-9w4dw\") pod \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\" (UID: \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\") "
Nov 03 20:36:37.652087 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.651984    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-extensions-socket-volume\") pod \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\" (UID: \"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254\") "
Nov 03 20:36:37.652399 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.652378    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-extensions-socket-volume" pod "d5dcfae1-34f9-4d34-b540-bc1dc9fa1254" (UID: "d5dcfae1-34f9-4d34-b540-bc1dc9fa1254"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:37.654390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.654364    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-kube-api-access-9w4dw" pod "d5dcfae1-34f9-4d34-b540-bc1dc9fa1254" (UID: "d5dcfae1-34f9-4d34-b540-bc1dc9fa1254"). InnerVolumeSpecName "kube-api-access-9w4dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:37.753199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.753161    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w4dw\" (UniqueName: \"kubernetes.io/projected/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-kube-api-access-9w4dw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:37.753199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:37.753194    5196 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254-extensions-socket-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:38.154732 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:38.154700    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:38.491438 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:38.491342    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm" event={"ID":"d5dcfae1-34f9-4d34-b540-bc1dc9fa1254","Type":"ContainerDied","Data":"1961c5156d90707cafb55f5d476b3bebc14dfcc64e79135c0c912f65ad1d42d3"}
Nov 03 20:36:38.491706 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:38.491685    5196 scope.go:122] "RemoveContainer" containerID="a341e8cb0f2d165d7b8c0e9a101475ad7404bf30e71f08af1f5a5ec4a255e425"
Nov 03 20:36:38.491901 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:38.491477    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"
Nov 03 20:36:38.693598 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:38.693554    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"]
Nov 03 20:36:38.741365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:38.741332    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vmfgm"]
Nov 03 20:36:40.107201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:40.107166    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dcfae1-34f9-4d34-b540-bc1dc9fa1254" path="/var/lib/kubelet/pods/d5dcfae1-34f9-4d34-b540-bc1dc9fa1254/volumes"
Nov 03 20:36:43.138394 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:43.138350    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:43.286795 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:43.286757    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:43.435813 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:43.435736    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:48.485001 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:36:48.484963    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:36:57.333013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.332925    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:36:57.333489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.333469    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d5dcfae1-34f9-4d34-b540-bc1dc9fa1254" containerName="manager"
Nov 03 20:36:57.333537 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.333493    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dcfae1-34f9-4d34-b540-bc1dc9fa1254" containerName="manager"
Nov 03 20:36:57.333595 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.333585    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dcfae1-34f9-4d34-b540-bc1dc9fa1254" containerName="manager"
Nov 03 20:36:57.336853 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.336836    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:57.388862 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.388828    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:36:57.414224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.414183    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9830db65-bd33-400b-805c-d8b27fbaf442-config-file\") pod \"limitador-limitador-7c9b87d574-h2grx\" (UID: \"9830db65-bd33-400b-805c-d8b27fbaf442\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:57.414404 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.414327    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94j4\" (UniqueName: \"kubernetes.io/projected/9830db65-bd33-400b-805c-d8b27fbaf442-kube-api-access-b94j4\") pod \"limitador-limitador-7c9b87d574-h2grx\" (UID: \"9830db65-bd33-400b-805c-d8b27fbaf442\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:57.515486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.515447    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9830db65-bd33-400b-805c-d8b27fbaf442-config-file\") pod \"limitador-limitador-7c9b87d574-h2grx\" (UID: \"9830db65-bd33-400b-805c-d8b27fbaf442\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:57.515677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.515547    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94j4\" (UniqueName: \"kubernetes.io/projected/9830db65-bd33-400b-805c-d8b27fbaf442-kube-api-access-b94j4\") pod \"limitador-limitador-7c9b87d574-h2grx\" (UID: \"9830db65-bd33-400b-805c-d8b27fbaf442\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:57.516075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.516050    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9830db65-bd33-400b-805c-d8b27fbaf442-config-file\") pod \"limitador-limitador-7c9b87d574-h2grx\" (UID: \"9830db65-bd33-400b-805c-d8b27fbaf442\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:57.569502 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.569468    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94j4\" (UniqueName: \"kubernetes.io/projected/9830db65-bd33-400b-805c-d8b27fbaf442-kube-api-access-b94j4\") pod \"limitador-limitador-7c9b87d574-h2grx\" (UID: \"9830db65-bd33-400b-805c-d8b27fbaf442\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:57.650437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.650350    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:57.651272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.651246    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:36:57.709656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.709625    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:57.900930 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:57.900843    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:58.024775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.024737    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:36:58.030837 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:36:58.030802    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9830db65_bd33_400b_805c_d8b27fbaf442.slice/crio-4f570825cbfa65f9fb02b2decc98255b48144e582bf94016cd9add5e3c1e12c4 WatchSource:0}: Error finding container 4f570825cbfa65f9fb02b2decc98255b48144e582bf94016cd9add5e3c1e12c4: Status 404 returned error can't find the container with id 4f570825cbfa65f9fb02b2decc98255b48144e582bf94016cd9add5e3c1e12c4
Nov 03 20:36:58.616141 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.616087    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx" event={"ID":"9830db65-bd33-400b-805c-d8b27fbaf442","Type":"ContainerStarted","Data":"f76a90574d5f9a4a2e7c8cc5c26ddbd5d091eab5901ed538da8024e2009d4858"}
Nov 03 20:36:58.616575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.616159    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx" event={"ID":"9830db65-bd33-400b-805c-d8b27fbaf442","Type":"ContainerStarted","Data":"4f570825cbfa65f9fb02b2decc98255b48144e582bf94016cd9add5e3c1e12c4"}
Nov 03 20:36:58.616575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.616200    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:36:58.723919 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.723874    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx" podStartSLOduration=1.723861007 podStartE2EDuration="1.723861007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:58.723262963 +0000 UTC m=+458.727221678" watchObservedRunningTime="2026-11-03 20:36:58.723861007 +0000 UTC m=+458.727819731"
Nov 03 20:36:58.869009 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.868932    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-q9k4v"]
Nov 03 20:36:58.872518 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.872500    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-q9k4v"
Nov 03 20:36:58.921752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.921714    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-q9k4v"]
Nov 03 20:36:58.921908 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.921736    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-td6zp\""
Nov 03 20:36:58.927332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:58.927302    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgtjg\" (UniqueName: \"kubernetes.io/projected/ac278a23-a0eb-4fc8-82bf-fb753afb9a94-kube-api-access-qgtjg\") pod \"authorino-57f7f8659f-q9k4v\" (UID: \"ac278a23-a0eb-4fc8-82bf-fb753afb9a94\") " pod="kuadrant-system/authorino-57f7f8659f-q9k4v"
Nov 03 20:36:59.028565 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:59.028525    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgtjg\" (UniqueName: \"kubernetes.io/projected/ac278a23-a0eb-4fc8-82bf-fb753afb9a94-kube-api-access-qgtjg\") pod \"authorino-57f7f8659f-q9k4v\" (UID: \"ac278a23-a0eb-4fc8-82bf-fb753afb9a94\") " pod="kuadrant-system/authorino-57f7f8659f-q9k4v"
Nov 03 20:36:59.058972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:59.058935    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:59.082372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:59.082334    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgtjg\" (UniqueName: \"kubernetes.io/projected/ac278a23-a0eb-4fc8-82bf-fb753afb9a94-kube-api-access-qgtjg\") pod \"authorino-57f7f8659f-q9k4v\" (UID: \"ac278a23-a0eb-4fc8-82bf-fb753afb9a94\") " pod="kuadrant-system/authorino-57f7f8659f-q9k4v"
Nov 03 20:36:59.202599 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:59.202514    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-q9k4v"
Nov 03 20:36:59.206903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:59.206871    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:59.571346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:59.571310    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-q9k4v"]
Nov 03 20:36:59.577441 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:36:59.577398    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac278a23_a0eb_4fc8_82bf_fb753afb9a94.slice/crio-093198d7033c44031dd2f9b14e22454dded4f83e6894bfc858a7d4955be5dff9 WatchSource:0}: Error finding container 093198d7033c44031dd2f9b14e22454dded4f83e6894bfc858a7d4955be5dff9: Status 404 returned error can't find the container with id 093198d7033c44031dd2f9b14e22454dded4f83e6894bfc858a7d4955be5dff9
Nov 03 20:36:59.624107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:36:59.624069    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-q9k4v" event={"ID":"ac278a23-a0eb-4fc8-82bf-fb753afb9a94","Type":"ContainerStarted","Data":"093198d7033c44031dd2f9b14e22454dded4f83e6894bfc858a7d4955be5dff9"}
Nov 03 20:37:00.436138 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:00.436086    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9n5jh"]
Nov 03 20:37:00.440732 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:00.440707    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh"
Nov 03 20:37:00.491181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:00.491152    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9n5jh"]
Nov 03 20:37:00.541642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:00.541585    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs6rl\" (UniqueName: \"kubernetes.io/projected/c7d8b932-c50b-41f2-9743-a2670c8baefb-kube-api-access-cs6rl\") pod \"authorino-56c6c9ddd5-9n5jh\" (UID: \"c7d8b932-c50b-41f2-9743-a2670c8baefb\") " pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh"
Nov 03 20:37:00.642817 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:00.642786    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs6rl\" (UniqueName: \"kubernetes.io/projected/c7d8b932-c50b-41f2-9743-a2670c8baefb-kube-api-access-cs6rl\") pod \"authorino-56c6c9ddd5-9n5jh\" (UID: \"c7d8b932-c50b-41f2-9743-a2670c8baefb\") " pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh"
Nov 03 20:37:00.696902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:00.696821    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs6rl\" (UniqueName: \"kubernetes.io/projected/c7d8b932-c50b-41f2-9743-a2670c8baefb-kube-api-access-cs6rl\") pod \"authorino-56c6c9ddd5-9n5jh\" (UID: \"c7d8b932-c50b-41f2-9743-a2670c8baefb\") " pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh"
Nov 03 20:37:00.752606 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:00.752557    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh"
Nov 03 20:37:01.122872 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:01.122830    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9n5jh"]
Nov 03 20:37:01.130165 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:37:01.130127    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7d8b932_c50b_41f2_9743_a2670c8baefb.slice/crio-a508e61ddbff74a8c72ea471a51deb3d41eb0100d8a669606f4b98e806f45f9d WatchSource:0}: Error finding container a508e61ddbff74a8c72ea471a51deb3d41eb0100d8a669606f4b98e806f45f9d: Status 404 returned error can't find the container with id a508e61ddbff74a8c72ea471a51deb3d41eb0100d8a669606f4b98e806f45f9d
Nov 03 20:37:01.637001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:01.636959    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh" event={"ID":"c7d8b932-c50b-41f2-9743-a2670c8baefb","Type":"ContainerStarted","Data":"a508e61ddbff74a8c72ea471a51deb3d41eb0100d8a669606f4b98e806f45f9d"}
Nov 03 20:37:03.651037 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:03.650998    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-q9k4v" event={"ID":"ac278a23-a0eb-4fc8-82bf-fb753afb9a94","Type":"ContainerStarted","Data":"090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2"}
Nov 03 20:37:03.652763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:03.652735    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh" event={"ID":"c7d8b932-c50b-41f2-9743-a2670c8baefb","Type":"ContainerStarted","Data":"bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c"}
Nov 03 20:37:03.757762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:03.757658    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-q9k4v" podStartSLOduration=2.192737462 podStartE2EDuration="5.757643378s" totalImagesPullingTime="3.564905916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:03.756076717 +0000 UTC m=+463.760035442" watchObservedRunningTime="2026-11-03 20:37:03.757643378 +0000 UTC m=+463.761602102"
Nov 03 20:37:03.861951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:03.861892    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh" podStartSLOduration=1.498350071 podStartE2EDuration="3.861876789s" totalImagesPullingTime="2.363526718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:03.860886751 +0000 UTC m=+463.864845483" watchObservedRunningTime="2026-11-03 20:37:03.861876789 +0000 UTC m=+463.865835512"
Nov 03 20:37:03.882502 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:03.882462    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-q9k4v"]
Nov 03 20:37:05.663200 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:05.663162    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-q9k4v" podUID="ac278a23-a0eb-4fc8-82bf-fb753afb9a94" containerName="authorino" containerID="cri-o://090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2" gracePeriod=30
Nov 03 20:37:05.914902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:05.914839    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-q9k4v"
Nov 03 20:37:05.995017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:05.994983    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac278a23-a0eb-4fc8-82bf-fb753afb9a94-kube-api-access-qgtjg\" (UniqueName: \"kubernetes.io/projected/ac278a23-a0eb-4fc8-82bf-fb753afb9a94-kube-api-access-qgtjg\") pod \"ac278a23-a0eb-4fc8-82bf-fb753afb9a94\" (UID: \"ac278a23-a0eb-4fc8-82bf-fb753afb9a94\") "
Nov 03 20:37:05.997214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:05.997185    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac278a23-a0eb-4fc8-82bf-fb753afb9a94-kube-api-access-qgtjg" pod "ac278a23-a0eb-4fc8-82bf-fb753afb9a94" (UID: "ac278a23-a0eb-4fc8-82bf-fb753afb9a94"). InnerVolumeSpecName "kube-api-access-qgtjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:06.095988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.095939    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgtjg\" (UniqueName: \"kubernetes.io/projected/ac278a23-a0eb-4fc8-82bf-fb753afb9a94-kube-api-access-qgtjg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:06.669466 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.669429    5196 generic.go:356] "Generic (PLEG): container finished" podID="ac278a23-a0eb-4fc8-82bf-fb753afb9a94" containerID="090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2" exitCode=0
Nov 03 20:37:06.669894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.669477    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-q9k4v"
Nov 03 20:37:06.669894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.669522    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-q9k4v" event={"ID":"ac278a23-a0eb-4fc8-82bf-fb753afb9a94","Type":"ContainerDied","Data":"090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2"}
Nov 03 20:37:06.669894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.669567    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-q9k4v" event={"ID":"ac278a23-a0eb-4fc8-82bf-fb753afb9a94","Type":"ContainerDied","Data":"093198d7033c44031dd2f9b14e22454dded4f83e6894bfc858a7d4955be5dff9"}
Nov 03 20:37:06.669894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.669587    5196 scope.go:122] "RemoveContainer" containerID="090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2"
Nov 03 20:37:06.678691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.678664    5196 scope.go:122] "RemoveContainer" containerID="090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2"
Nov 03 20:37:06.679016 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:37:06.678990    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2\": container with ID starting with 090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2 not found: ID does not exist" containerID="090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2"
Nov 03 20:37:06.679076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.679031    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2"} err="failed to get container status \"090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2\": rpc error: code = NotFound desc = could not find container \"090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2\": container with ID starting with 090c55278f09913720fb38b3a5e1a1fb184503a988c579918810a466051b40f2 not found: ID does not exist"
Nov 03 20:37:06.826505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.826470    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-q9k4v"]
Nov 03 20:37:06.830570 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:06.830546    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-q9k4v"]
Nov 03 20:37:08.107107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:08.107060    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac278a23-a0eb-4fc8-82bf-fb753afb9a94" path="/var/lib/kubelet/pods/ac278a23-a0eb-4fc8-82bf-fb753afb9a94/volumes"
Nov 03 20:37:09.625683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:09.625654    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-h2grx"
Nov 03 20:37:09.781210 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:09.781179    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"]
Nov 03 20:37:09.781399 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:09.781371    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt" podUID="04366c39-0027-4379-92be-115a0979a562" containerName="limitador" containerID="cri-o://2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9" gracePeriod=30
Nov 03 20:37:10.334365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.334339    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:37:10.432780 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.432700    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04366c39-0027-4379-92be-115a0979a562-config-file\" (UniqueName: \"kubernetes.io/configmap/04366c39-0027-4379-92be-115a0979a562-config-file\") pod \"04366c39-0027-4379-92be-115a0979a562\" (UID: \"04366c39-0027-4379-92be-115a0979a562\") "
Nov 03 20:37:10.432935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.432789    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04366c39-0027-4379-92be-115a0979a562-kube-api-access-f49jh\" (UniqueName: \"kubernetes.io/projected/04366c39-0027-4379-92be-115a0979a562-kube-api-access-f49jh\") pod \"04366c39-0027-4379-92be-115a0979a562\" (UID: \"04366c39-0027-4379-92be-115a0979a562\") "
Nov 03 20:37:10.433142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.433075    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04366c39-0027-4379-92be-115a0979a562-config-file" pod "04366c39-0027-4379-92be-115a0979a562" (UID: "04366c39-0027-4379-92be-115a0979a562"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:10.435036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.435008    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04366c39-0027-4379-92be-115a0979a562-kube-api-access-f49jh" pod "04366c39-0027-4379-92be-115a0979a562" (UID: "04366c39-0027-4379-92be-115a0979a562"). InnerVolumeSpecName "kube-api-access-f49jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:10.534292 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.534254    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f49jh\" (UniqueName: \"kubernetes.io/projected/04366c39-0027-4379-92be-115a0979a562-kube-api-access-f49jh\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:10.534292 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.534285    5196 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04366c39-0027-4379-92be-115a0979a562-config-file\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:10.692879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.692793    5196 generic.go:356] "Generic (PLEG): container finished" podID="04366c39-0027-4379-92be-115a0979a562" containerID="2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9" exitCode=0
Nov 03 20:37:10.692879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.692840    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"
Nov 03 20:37:10.692879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.692854    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt" event={"ID":"04366c39-0027-4379-92be-115a0979a562","Type":"ContainerDied","Data":"2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9"}
Nov 03 20:37:10.693491 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.692906    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ttkgt" event={"ID":"04366c39-0027-4379-92be-115a0979a562","Type":"ContainerDied","Data":"af67a900f09bc763ba12b64df004da764bba6384b8de7af106cd29a83433199b"}
Nov 03 20:37:10.693491 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.692935    5196 scope.go:122] "RemoveContainer" containerID="2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9"
Nov 03 20:37:10.702829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.702804    5196 scope.go:122] "RemoveContainer" containerID="2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9"
Nov 03 20:37:10.703130 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:37:10.703108    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9\": container with ID starting with 2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9 not found: ID does not exist" containerID="2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9"
Nov 03 20:37:10.703214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.703140    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9"} err="failed to get container status \"2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9\": rpc error: code = NotFound desc = could not find container \"2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9\": container with ID starting with 2905e1bf8e50e1d19b0a2de1a3a62e346b2c2ed0bab674766e322839e5efa8d9 not found: ID does not exist"
Nov 03 20:37:10.848829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.848577    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"]
Nov 03 20:37:10.894738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:10.894696    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ttkgt"]
Nov 03 20:37:12.108309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:12.108275    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04366c39-0027-4379-92be-115a0979a562" path="/var/lib/kubelet/pods/04366c39-0027-4379-92be-115a0979a562/volumes"
Nov 03 20:37:16.608908 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:16.608872    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:16.789049 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:16.789010    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:17.271542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.271506    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"]
Nov 03 20:37:17.271965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.271948    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ac278a23-a0eb-4fc8-82bf-fb753afb9a94" containerName="authorino"
Nov 03 20:37:17.272058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.271969    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac278a23-a0eb-4fc8-82bf-fb753afb9a94" containerName="authorino"
Nov 03 20:37:17.272058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.271984    5196 cpu_manager.go:414] "containerMap: removing container" podUID="04366c39-0027-4379-92be-115a0979a562" containerName="limitador"
Nov 03 20:37:17.272058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.271991    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04366c39-0027-4379-92be-115a0979a562" containerName="limitador"
Nov 03 20:37:17.272233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.272158    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac278a23-a0eb-4fc8-82bf-fb753afb9a94" containerName="authorino"
Nov 03 20:37:17.272233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.272172    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="04366c39-0027-4379-92be-115a0979a562" containerName="limitador"
Nov 03 20:37:17.276546 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.276524    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:17.334183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.334144    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"]
Nov 03 20:37:17.376493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.376458    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-92dp9\""
Nov 03 20:37:17.376673 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.376469    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 20:37:17.376673 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.376470    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:37:17.376673 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.376487    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:37:17.387701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.387646    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:17.398705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.398667    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a269e6a3-f734-4a59-a71d-764f59259a36-data\") pod \"postgres-7cdbf6dc44-frjx5\" (UID: \"a269e6a3-f734-4a59-a71d-764f59259a36\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:17.398842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.398709    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwms\" (UniqueName: \"kubernetes.io/projected/a269e6a3-f734-4a59-a71d-764f59259a36-kube-api-access-5cwms\") pod \"postgres-7cdbf6dc44-frjx5\" (UID: \"a269e6a3-f734-4a59-a71d-764f59259a36\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:17.500176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.500130    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a269e6a3-f734-4a59-a71d-764f59259a36-data\") pod \"postgres-7cdbf6dc44-frjx5\" (UID: \"a269e6a3-f734-4a59-a71d-764f59259a36\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:17.500391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.500184    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwms\" (UniqueName: \"kubernetes.io/projected/a269e6a3-f734-4a59-a71d-764f59259a36-kube-api-access-5cwms\") pod \"postgres-7cdbf6dc44-frjx5\" (UID: \"a269e6a3-f734-4a59-a71d-764f59259a36\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:17.500625 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.500601    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a269e6a3-f734-4a59-a71d-764f59259a36-data\") pod \"postgres-7cdbf6dc44-frjx5\" (UID: \"a269e6a3-f734-4a59-a71d-764f59259a36\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:17.554740 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.554666    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwms\" (UniqueName: \"kubernetes.io/projected/a269e6a3-f734-4a59-a71d-764f59259a36-kube-api-access-5cwms\") pod \"postgres-7cdbf6dc44-frjx5\" (UID: \"a269e6a3-f734-4a59-a71d-764f59259a36\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:17.591691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.591656    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:17.977852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:17.977817    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"]
Nov 03 20:37:17.983702 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:37:17.983670    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda269e6a3_f734_4a59_a71d_764f59259a36.slice/crio-636b26cf5466ef61422aa06d87e22343efb32fa940d65e6c3928235c1ee0684a WatchSource:0}: Error finding container 636b26cf5466ef61422aa06d87e22343efb32fa940d65e6c3928235c1ee0684a: Status 404 returned error can't find the container with id 636b26cf5466ef61422aa06d87e22343efb32fa940d65e6c3928235c1ee0684a
Nov 03 20:37:18.485076 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:37:18.485041    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:37:18.739875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:18.739783    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5" event={"ID":"a269e6a3-f734-4a59-a71d-764f59259a36","Type":"ContainerStarted","Data":"636b26cf5466ef61422aa06d87e22343efb32fa940d65e6c3928235c1ee0684a"}
Nov 03 20:37:27.114083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:27.114046    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 20:37:27.796831 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:27.796790    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5" event={"ID":"a269e6a3-f734-4a59-a71d-764f59259a36","Type":"ContainerStarted","Data":"0f3b03ba37170c3a6fb9cb3db639a7e48c434ed58eb2f4b71a2e3a31f9d92717"}
Nov 03 20:37:27.797067 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:27.796858    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:27.904782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:27.904731    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5" podStartSLOduration=1.824502482 podStartE2EDuration="10.904716842s" totalImagesPullingTime="9.08021436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:27.90325143 +0000 UTC m=+487.907210154" watchObservedRunningTime="2026-11-03 20:37:27.904716842 +0000 UTC m=+487.908675565"
Nov 03 20:37:30.623987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:30.623943    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.143563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.143518    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.294873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.294833    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.446531 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.446439    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.604668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.604633    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.753455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.753362    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.902817 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.902789    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.936068 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.936035    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4jfx4"]
Nov 03 20:37:31.939568 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.939546    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:31.993053 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:31.993018    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4jfx4"]
Nov 03 20:37:32.036729 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.036645    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 20:37:32.036729 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.036682    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 20:37:32.036729 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.036682    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 20:37:32.036951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.036732    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-59nrb\""
Nov 03 20:37:32.039235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.039212    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkc6\" (UniqueName: \"kubernetes.io/projected/0e9476a2-3254-447f-bcfd-ffa8804e4d2f-kube-api-access-bfkc6\") pod \"kserve-controller-manager-776cc66d47-4jfx4\" (UID: \"0e9476a2-3254-447f-bcfd-ffa8804e4d2f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:32.039345 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.039255    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e9476a2-3254-447f-bcfd-ffa8804e4d2f-cert\") pod \"kserve-controller-manager-776cc66d47-4jfx4\" (UID: \"0e9476a2-3254-447f-bcfd-ffa8804e4d2f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:32.074662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.074624    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:32.140473 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.140442    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkc6\" (UniqueName: \"kubernetes.io/projected/0e9476a2-3254-447f-bcfd-ffa8804e4d2f-kube-api-access-bfkc6\") pod \"kserve-controller-manager-776cc66d47-4jfx4\" (UID: \"0e9476a2-3254-447f-bcfd-ffa8804e4d2f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:32.140657 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.140482    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e9476a2-3254-447f-bcfd-ffa8804e4d2f-cert\") pod \"kserve-controller-manager-776cc66d47-4jfx4\" (UID: \"0e9476a2-3254-447f-bcfd-ffa8804e4d2f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:32.143190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.143165    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e9476a2-3254-447f-bcfd-ffa8804e4d2f-cert\") pod \"kserve-controller-manager-776cc66d47-4jfx4\" (UID: \"0e9476a2-3254-447f-bcfd-ffa8804e4d2f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:32.194708 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.194673    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkc6\" (UniqueName: \"kubernetes.io/projected/0e9476a2-3254-447f-bcfd-ffa8804e4d2f-kube-api-access-bfkc6\") pod \"kserve-controller-manager-776cc66d47-4jfx4\" (UID: \"0e9476a2-3254-447f-bcfd-ffa8804e4d2f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:32.232921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.232887    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:32.249702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.249667    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-lkc7w"]
Nov 03 20:37:32.252345 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.252319    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:32.253797 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.253767    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.305119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.304418    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 20:37:32.305119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.304726    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6l2fp\""
Nov 03 20:37:32.307631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.307610    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 20:37:32.310310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.310235    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-lkc7w"]
Nov 03 20:37:32.341772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.341727    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd606fe-f716-464a-bbf3-6e3eec674b5b-cert\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.341927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.341823    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprd9\" (UniqueName: \"kubernetes.io/projected/9cd606fe-f716-464a-bbf3-6e3eec674b5b-kube-api-access-vprd9\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.341992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.341948    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd606fe-f716-464a-bbf3-6e3eec674b5b-metrics-certs\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.442524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.442481    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd606fe-f716-464a-bbf3-6e3eec674b5b-cert\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.442677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.442579    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprd9\" (UniqueName: \"kubernetes.io/projected/9cd606fe-f716-464a-bbf3-6e3eec674b5b-kube-api-access-vprd9\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.442677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.442655    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd606fe-f716-464a-bbf3-6e3eec674b5b-metrics-certs\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.445384 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.445347    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd606fe-f716-464a-bbf3-6e3eec674b5b-metrics-certs\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.445505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.445347    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd606fe-f716-464a-bbf3-6e3eec674b5b-cert\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.496489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.496456    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprd9\" (UniqueName: \"kubernetes.io/projected/9cd606fe-f716-464a-bbf3-6e3eec674b5b-kube-api-access-vprd9\") pod \"odh-model-controller-769875db56-lkc7w\" (UID: \"9cd606fe-f716-464a-bbf3-6e3eec674b5b\") " pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.583252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.583215    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:32.623359 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.623312    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4jfx4"]
Nov 03 20:37:32.630022 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:37:32.629991    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9476a2_3254_447f_bcfd_ffa8804e4d2f.slice/crio-ce599415475c9cf06e6c4aca869e183888def0927c2c92fe7a8d8959120dc92b WatchSource:0}: Error finding container ce599415475c9cf06e6c4aca869e183888def0927c2c92fe7a8d8959120dc92b: Status 404 returned error can't find the container with id ce599415475c9cf06e6c4aca869e183888def0927c2c92fe7a8d8959120dc92b
Nov 03 20:37:32.825075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.825040    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4" event={"ID":"0e9476a2-3254-447f-bcfd-ffa8804e4d2f","Type":"ContainerStarted","Data":"ce599415475c9cf06e6c4aca869e183888def0927c2c92fe7a8d8959120dc92b"}
Nov 03 20:37:32.955289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:32.955207    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-lkc7w"]
Nov 03 20:37:32.961578 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:37:32.961549    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd606fe_f716_464a_bbf3_6e3eec674b5b.slice/crio-00abc9f6251bc6dbc6ca6190c17b408b23c837747b08fe88f3fcf47deae89fd1 WatchSource:0}: Error finding container 00abc9f6251bc6dbc6ca6190c17b408b23c837747b08fe88f3fcf47deae89fd1: Status 404 returned error can't find the container with id 00abc9f6251bc6dbc6ca6190c17b408b23c837747b08fe88f3fcf47deae89fd1
Nov 03 20:37:33.244979 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:33.244895    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:33.834769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:33.834729    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-lkc7w" event={"ID":"9cd606fe-f716-464a-bbf3-6e3eec674b5b","Type":"ContainerStarted","Data":"00abc9f6251bc6dbc6ca6190c17b408b23c837747b08fe88f3fcf47deae89fd1"}
Nov 03 20:37:33.840780 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:33.840755    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-frjx5"
Nov 03 20:37:37.925294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:37.925252    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.118294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.118262    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.270908 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.270819    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.418157 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.418125    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.558835 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.558756    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.708496 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.708459    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.858276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.858239    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.869230 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.869186    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4" event={"ID":"0e9476a2-3254-447f-bcfd-ffa8804e4d2f","Type":"ContainerStarted","Data":"f04d4de8d36c8bf4f33b9699e7ab0ae8ee4fe2afe848840080651bf213439d5e"}
Nov 03 20:37:38.869425 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.869314    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:37:38.981142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:38.981062    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4" podStartSLOduration=2.225021992 podStartE2EDuration="7.981047508s" totalImagesPullingTime="5.756025516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:38.980535898 +0000 UTC m=+498.984494622" watchObservedRunningTime="2026-11-03 20:37:38.981047508 +0000 UTC m=+498.985006232"
Nov 03 20:37:39.013045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:39.013007    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.516387 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.516348    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.557368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.557335    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4mh8r"]
Nov 03 20:37:40.561388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.561357    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.644767 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.644736    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:37:40.657895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.657861    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 20:37:40.658117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.657943    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 20:37:40.658117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.658030    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrkqw\""
Nov 03 20:37:40.661481 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.661448    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4mh8r"]
Nov 03 20:37:40.713047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.713011    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.723083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.723053    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-metrics-certs\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.723083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.723087    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hqb\" (UniqueName: \"kubernetes.io/projected/96cc723a-7eb4-4b73-ac24-c5acba964341-kube-api-access-q5hqb\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.723324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.723168    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-webhook-cert\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.823897 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.823861    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-metrics-certs\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.824194 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.823904    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hqb\" (UniqueName: \"kubernetes.io/projected/96cc723a-7eb4-4b73-ac24-c5acba964341-kube-api-access-q5hqb\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.824194 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.823955    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-webhook-cert\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.826703 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.826658    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-metrics-certs\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.826825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.826769    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-webhook-cert\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.862677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.862621    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.878449 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.878420    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hqb\" (UniqueName: \"kubernetes.io/projected/96cc723a-7eb4-4b73-ac24-c5acba964341-kube-api-access-q5hqb\") pod \"maas-controller-579cf7c6d-4mh8r\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") " pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:40.958068 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.958028    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8f44875fd-nh9tp"]
Nov 03 20:37:40.961918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:40.961894    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.016202 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.015984    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:41.016918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.016897    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8f44875fd-nh9tp"]
Nov 03 20:37:41.025668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.025636    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-webhook-cert\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.025811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.025672    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-metrics-certs\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.025811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.025781    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whd4v\" (UniqueName: \"kubernetes.io/projected/9626020c-90dc-43d9-b77f-d512892db1ce-kube-api-access-whd4v\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.126444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.126353    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-webhook-cert\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.126702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.126679    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-metrics-certs\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.126906 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.126888    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whd4v\" (UniqueName: \"kubernetes.io/projected/9626020c-90dc-43d9-b77f-d512892db1ce-kube-api-access-whd4v\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.129710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.129676    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-metrics-certs\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.129710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.129698    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-webhook-cert\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.175435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.175398    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:41.181043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.181022    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whd4v\" (UniqueName: \"kubernetes.io/projected/9626020c-90dc-43d9-b77f-d512892db1ce-kube-api-access-whd4v\") pod \"maas-controller-8f44875fd-nh9tp\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") " pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.275183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.275151    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:41.551332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.551250    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4mh8r"]
Nov 03 20:37:41.557333 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:37:41.557297    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cc723a_7eb4_4b73_ac24_c5acba964341.slice/crio-14266e7608b80453a0cc8bb77b2cc7fa0d9ce4b15ab84a457ccae7b8cece3290 WatchSource:0}: Error finding container 14266e7608b80453a0cc8bb77b2cc7fa0d9ce4b15ab84a457ccae7b8cece3290: Status 404 returned error can't find the container with id 14266e7608b80453a0cc8bb77b2cc7fa0d9ce4b15ab84a457ccae7b8cece3290
Nov 03 20:37:41.663854 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.663818    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8f44875fd-nh9tp"]
Nov 03 20:37:41.671164 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:37:41.671134    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9626020c_90dc_43d9_b77f_d512892db1ce.slice/crio-3be6948e671df043b9ea68bb2e2576dc0b5753ca22283a4961dfffa51184a471 WatchSource:0}: Error finding container 3be6948e671df043b9ea68bb2e2576dc0b5753ca22283a4961dfffa51184a471: Status 404 returned error can't find the container with id 3be6948e671df043b9ea68bb2e2576dc0b5753ca22283a4961dfffa51184a471
Nov 03 20:37:41.891289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.891244    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4mh8r" event={"ID":"96cc723a-7eb4-4b73-ac24-c5acba964341","Type":"ContainerStarted","Data":"14266e7608b80453a0cc8bb77b2cc7fa0d9ce4b15ab84a457ccae7b8cece3290"}
Nov 03 20:37:41.892816 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:41.892776    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-nh9tp" event={"ID":"9626020c-90dc-43d9-b77f-d512892db1ce","Type":"ContainerStarted","Data":"3be6948e671df043b9ea68bb2e2576dc0b5753ca22283a4961dfffa51184a471"}
Nov 03 20:37:42.901437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:42.901398    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-lkc7w" event={"ID":"9cd606fe-f716-464a-bbf3-6e3eec674b5b","Type":"ContainerStarted","Data":"fb0a1c8453bbb8817db047c9da5899a8b7ca918c6137c0f1ab6afe8a005d0a7f"}
Nov 03 20:37:42.901860 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:42.901680    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:43.011804 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:43.011740    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-lkc7w" podStartSLOduration=2.007432808 podStartE2EDuration="11.011720197s" totalImagesPullingTime="9.004287389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:43.008480552 +0000 UTC m=+503.012439268" watchObservedRunningTime="2026-11-03 20:37:43.011720197 +0000 UTC m=+503.015678922"
Nov 03 20:37:47.341139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:47.341085    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:47.539500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:47.539463    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:47.932189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:47.932146    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-nh9tp" event={"ID":"9626020c-90dc-43d9-b77f-d512892db1ce","Type":"ContainerStarted","Data":"75557eb4230eb616b92797f6a2469851e48a84204705a0534fb4712145d4cf01"}
Nov 03 20:37:47.932407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:47.932208    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:47.933750 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:47.933721    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4mh8r" event={"ID":"96cc723a-7eb4-4b73-ac24-c5acba964341","Type":"ContainerStarted","Data":"16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a"}
Nov 03 20:37:47.933870 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:47.933767    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:48.082706 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:48.082638    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8f44875fd-nh9tp" podStartSLOduration=2.805999018 podStartE2EDuration="8.082623009s" totalImagesPullingTime="5.276623991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:48.080320759 +0000 UTC m=+508.084279510" watchObservedRunningTime="2026-11-03 20:37:48.082623009 +0000 UTC m=+508.086581736"
Nov 03 20:37:48.191792 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:48.191661    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-4mh8r" podStartSLOduration=2.842818717 podStartE2EDuration="8.191640098s" totalImagesPullingTime="5.348821381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:48.188735303 +0000 UTC m=+508.192694055" watchObservedRunningTime="2026-11-03 20:37:48.191640098 +0000 UTC m=+508.195598855"
Nov 03 20:37:48.486254 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:37:48.486173    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:37:50.429588 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.429546    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"]
Nov 03 20:37:50.433642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.433618    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.508753 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.508718    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.508952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.508768    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7h4c\" (UniqueName: \"kubernetes.io/projected/b1a4914f-99fa-492e-9e2a-716fe58ab628-kube-api-access-w7h4c\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.508952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.508840    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.508952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.508896    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1a4914f-99fa-492e-9e2a-716fe58ab628-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.530769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.530733    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 20:37:50.530975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.530830    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-bjkgk\""
Nov 03 20:37:50.530975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.530842    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 20:37:50.530975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.530956    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 20:37:50.533186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.533161    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"]
Nov 03 20:37:50.597565 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.597529    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:50.609619 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.609593    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1a4914f-99fa-492e-9e2a-716fe58ab628-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.609733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.609653    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.609733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.609688    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7h4c\" (UniqueName: \"kubernetes.io/projected/b1a4914f-99fa-492e-9e2a-716fe58ab628-kube-api-access-w7h4c\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.609733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.609727    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.609834 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:37:50.609750    5196 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 20:37:50.609834 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:37:50.609806    5196 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 03 20:37:50.609834 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:37:50.609823    5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a4914f-99fa-492e-9e2a-716fe58ab628-tls-cert podName:b1a4914f-99fa-492e-9e2a-716fe58ab628 nodeName:}" failed. No retries permitted until 2026-11-03 20:37:51.109806043 +0000 UTC m=+511.113764762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/b1a4914f-99fa-492e-9e2a-716fe58ab628-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-7vlp8" (UID: "b1a4914f-99fa-492e-9e2a-716fe58ab628") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 20:37:50.609947 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:37:50.609857    5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-proxy-source podName:b1a4914f-99fa-492e-9e2a-716fe58ab628 nodeName:}" failed. No retries permitted until 2026-11-03 20:37:51.109845374 +0000 UTC m=+511.113804076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-7vlp8" (UID: "b1a4914f-99fa-492e-9e2a-716fe58ab628") : configmap "usage-tenancy-proxy-source" not found
Nov 03 20:37:50.610401 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.610382    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.666948 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.666918    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7h4c\" (UniqueName: \"kubernetes.io/projected/b1a4914f-99fa-492e-9e2a-716fe58ab628-kube-api-access-w7h4c\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:50.793967 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:50.793872    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:51.115036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:51.115003    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:51.115288 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:51.115081    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1a4914f-99fa-492e-9e2a-716fe58ab628-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:51.115937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:51.115909    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b1a4914f-99fa-492e-9e2a-716fe58ab628-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:51.117600 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:51.117582    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1a4914f-99fa-492e-9e2a-716fe58ab628-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7vlp8\" (UID: \"b1a4914f-99fa-492e-9e2a-716fe58ab628\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:51.346742 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:51.346710    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:37:51.721544 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:51.721462    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"]
Nov 03 20:37:51.728451 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:37:51.728423    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a4914f_99fa_492e_9e2a_716fe58ab628.slice/crio-6153b10120bfecd1f84e1be0688f27e2741e7571faec9e9eb63801b581002052 WatchSource:0}: Error finding container 6153b10120bfecd1f84e1be0688f27e2741e7571faec9e9eb63801b581002052: Status 404 returned error can't find the container with id 6153b10120bfecd1f84e1be0688f27e2741e7571faec9e9eb63801b581002052
Nov 03 20:37:51.956203 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:51.956165    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8" event={"ID":"b1a4914f-99fa-492e-9e2a-716fe58ab628","Type":"ContainerStarted","Data":"6153b10120bfecd1f84e1be0688f27e2741e7571faec9e9eb63801b581002052"}
Nov 03 20:37:53.912158 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:53.912087    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-lkc7w"
Nov 03 20:37:58.941701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:58.941664    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:58.942806 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:58.942788    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 20:37:59.191273 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.191187    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4mh8r"]
Nov 03 20:37:59.191964 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.191745    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-4mh8r" podUID="96cc723a-7eb4-4b73-ac24-c5acba964341" containerName="manager" containerID="cri-o://16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a" gracePeriod=30
Nov 03 20:37:59.556071 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.556037    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:37:59.608346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.608301    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-webhook-cert\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-webhook-cert\") pod \"96cc723a-7eb4-4b73-ac24-c5acba964341\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") "
Nov 03 20:37:59.608558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.608402    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cc723a-7eb4-4b73-ac24-c5acba964341-kube-api-access-q5hqb\" (UniqueName: \"kubernetes.io/projected/96cc723a-7eb4-4b73-ac24-c5acba964341-kube-api-access-q5hqb\") pod \"96cc723a-7eb4-4b73-ac24-c5acba964341\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") "
Nov 03 20:37:59.608558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.608451    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-metrics-certs\") pod \"96cc723a-7eb4-4b73-ac24-c5acba964341\" (UID: \"96cc723a-7eb4-4b73-ac24-c5acba964341\") "
Nov 03 20:37:59.611165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.611132    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cc723a-7eb4-4b73-ac24-c5acba964341-kube-api-access-q5hqb" pod "96cc723a-7eb4-4b73-ac24-c5acba964341" (UID: "96cc723a-7eb4-4b73-ac24-c5acba964341"). InnerVolumeSpecName "kube-api-access-q5hqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:59.611395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.611351    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-webhook-cert" pod "96cc723a-7eb4-4b73-ac24-c5acba964341" (UID: "96cc723a-7eb4-4b73-ac24-c5acba964341"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:59.611493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.611366    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-metrics-certs" pod "96cc723a-7eb4-4b73-ac24-c5acba964341" (UID: "96cc723a-7eb4-4b73-ac24-c5acba964341"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:59.709834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.709793    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5hqb\" (UniqueName: \"kubernetes.io/projected/96cc723a-7eb4-4b73-ac24-c5acba964341-kube-api-access-q5hqb\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:59.709834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.709828    5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-metrics-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:59.709834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:37:59.709838    5196 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96cc723a-7eb4-4b73-ac24-c5acba964341-webhook-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:00.015073 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.015034    5196 generic.go:356] "Generic (PLEG): container finished" podID="96cc723a-7eb4-4b73-ac24-c5acba964341" containerID="16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a" exitCode=0
Nov 03 20:38:00.015508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.015084    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4mh8r"
Nov 03 20:38:00.015508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.015125    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4mh8r" event={"ID":"96cc723a-7eb4-4b73-ac24-c5acba964341","Type":"ContainerDied","Data":"16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a"}
Nov 03 20:38:00.015508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.015184    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4mh8r" event={"ID":"96cc723a-7eb4-4b73-ac24-c5acba964341","Type":"ContainerDied","Data":"14266e7608b80453a0cc8bb77b2cc7fa0d9ce4b15ab84a457ccae7b8cece3290"}
Nov 03 20:38:00.015508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.015213    5196 scope.go:122] "RemoveContainer" containerID="16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a"
Nov 03 20:38:00.025907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.025888    5196 scope.go:122] "RemoveContainer" containerID="16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a"
Nov 03 20:38:00.026358 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:38:00.026326    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a\": container with ID starting with 16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a not found: ID does not exist" containerID="16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a"
Nov 03 20:38:00.026477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.026370    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a"} err="failed to get container status \"16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a\": rpc error: code = NotFound desc = could not find container \"16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a\": container with ID starting with 16d8251681c122976182dcb7018e9f20241282f39546941472fb04bc0616c88a not found: ID does not exist"
Nov 03 20:38:00.170027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.169986    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4mh8r"]
Nov 03 20:38:00.217348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:00.217310    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4mh8r"]
Nov 03 20:38:02.112004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:02.111966    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cc723a-7eb4-4b73-ac24-c5acba964341" path="/var/lib/kubelet/pods/96cc723a-7eb4-4b73-ac24-c5acba964341/volumes"
Nov 03 20:38:09.881346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:09.881311    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4jfx4"
Nov 03 20:38:16.122520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:16.122487    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8" event={"ID":"b1a4914f-99fa-492e-9e2a-716fe58ab628","Type":"ContainerStarted","Data":"ba3c09a0bb78ac571350d00d7d263453c7e48e4c75170086e5be28c1ec501998"}
Nov 03 20:38:16.122932 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:16.122539    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:38:16.231399 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:16.231336    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8" podStartSLOduration=2.259774405 podStartE2EDuration="26.231317847s" totalImagesPullingTime="23.971543442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:16.230502499 +0000 UTC m=+536.234461243" watchObservedRunningTime="2026-11-03 20:38:16.231317847 +0000 UTC m=+536.235276572"
Nov 03 20:38:18.483487 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:38:18.483446    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:38:22.780480 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:22.780442    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:22.942319 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:22.942280    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:24.895221 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:24.895129    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:25.092995 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:25.092959    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:25.239489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:25.239396    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:27.141454 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:27.141427    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7vlp8"
Nov 03 20:38:31.360297 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:31.360262    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:38:36.523333 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.523288    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:36.716290 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.716258    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:36.780874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.780794    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9g6sz"]
Nov 03 20:38:36.781305 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.781290    5196 cpu_manager.go:414] "containerMap: removing container" podUID="96cc723a-7eb4-4b73-ac24-c5acba964341" containerName="manager"
Nov 03 20:38:36.781364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.781307    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cc723a-7eb4-4b73-ac24-c5acba964341" containerName="manager"
Nov 03 20:38:36.781417 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.781406    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cc723a-7eb4-4b73-ac24-c5acba964341" containerName="manager"
Nov 03 20:38:36.874734 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.874697    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9g6sz"]
Nov 03 20:38:36.874914 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.874801    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9g6sz"
Nov 03 20:38:36.974924 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:36.974890    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2qg\" (UniqueName: \"kubernetes.io/projected/faa678f5-63c1-42ea-bc4c-92d862ed2946-kube-api-access-fz2qg\") pod \"authorino-6f6944bcdd-9g6sz\" (UID: \"faa678f5-63c1-42ea-bc4c-92d862ed2946\") " pod="kuadrant-system/authorino-6f6944bcdd-9g6sz"
Nov 03 20:38:37.075988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.075946    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2qg\" (UniqueName: \"kubernetes.io/projected/faa678f5-63c1-42ea-bc4c-92d862ed2946-kube-api-access-fz2qg\") pod \"authorino-6f6944bcdd-9g6sz\" (UID: \"faa678f5-63c1-42ea-bc4c-92d862ed2946\") " pod="kuadrant-system/authorino-6f6944bcdd-9g6sz"
Nov 03 20:38:37.139811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.134802    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9g6sz"]
Nov 03 20:38:37.139811 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:38:37.137543    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fz2qg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-9g6sz" podUID="faa678f5-63c1-42ea-bc4c-92d862ed2946"
Nov 03 20:38:37.149442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.149408    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2qg\" (UniqueName: \"kubernetes.io/projected/faa678f5-63c1-42ea-bc4c-92d862ed2946-kube-api-access-fz2qg\") pod \"authorino-6f6944bcdd-9g6sz\" (UID: \"faa678f5-63c1-42ea-bc4c-92d862ed2946\") " pod="kuadrant-system/authorino-6f6944bcdd-9g6sz"
Nov 03 20:38:37.160971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.160932    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ff895fdf5-np9p9"]
Nov 03 20:38:37.165105 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.165060    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ff895fdf5-np9p9"
Nov 03 20:38:37.176636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.176600    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpnsw\" (UniqueName: \"kubernetes.io/projected/df82d96b-aa8c-47ce-bda1-8f261d18c7b6-kube-api-access-vpnsw\") pod \"authorino-5ff895fdf5-np9p9\" (UID: \"df82d96b-aa8c-47ce-bda1-8f261d18c7b6\") " pod="kuadrant-system/authorino-5ff895fdf5-np9p9"
Nov 03 20:38:37.216673 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.216641    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ff895fdf5-np9p9"]
Nov 03 20:38:37.272666 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.272639    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9g6sz"
Nov 03 20:38:37.277373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.277348    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpnsw\" (UniqueName: \"kubernetes.io/projected/df82d96b-aa8c-47ce-bda1-8f261d18c7b6-kube-api-access-vpnsw\") pod \"authorino-5ff895fdf5-np9p9\" (UID: \"df82d96b-aa8c-47ce-bda1-8f261d18c7b6\") " pod="kuadrant-system/authorino-5ff895fdf5-np9p9"
Nov 03 20:38:37.282300 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.282279    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9g6sz"
Nov 03 20:38:37.331324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.331246    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpnsw\" (UniqueName: \"kubernetes.io/projected/df82d96b-aa8c-47ce-bda1-8f261d18c7b6-kube-api-access-vpnsw\") pod \"authorino-5ff895fdf5-np9p9\" (UID: \"df82d96b-aa8c-47ce-bda1-8f261d18c7b6\") " pod="kuadrant-system/authorino-5ff895fdf5-np9p9"
Nov 03 20:38:37.378066 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.378034    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa678f5-63c1-42ea-bc4c-92d862ed2946-kube-api-access-fz2qg\" (UniqueName: \"kubernetes.io/projected/faa678f5-63c1-42ea-bc4c-92d862ed2946-kube-api-access-fz2qg\") pod \"faa678f5-63c1-42ea-bc4c-92d862ed2946\" (UID: \"faa678f5-63c1-42ea-bc4c-92d862ed2946\") "
Nov 03 20:38:37.380437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.380400    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa678f5-63c1-42ea-bc4c-92d862ed2946-kube-api-access-fz2qg" pod "faa678f5-63c1-42ea-bc4c-92d862ed2946" (UID: "faa678f5-63c1-42ea-bc4c-92d862ed2946"). InnerVolumeSpecName "kube-api-access-fz2qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:37.479178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.479145    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz2qg\" (UniqueName: \"kubernetes.io/projected/faa678f5-63c1-42ea-bc4c-92d862ed2946-kube-api-access-fz2qg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:37.479350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.479218    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ff895fdf5-np9p9"
Nov 03 20:38:37.616620 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.616531    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ff895fdf5-np9p9"]
Nov 03 20:38:37.638952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.638917    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f7884ff45-9zq5j"]
Nov 03 20:38:37.642557 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.642535    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:38:37.681619 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.681579    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66a3239c-4083-424d-b164-109bb402bcfe-tls-cert\") pod \"authorino-7f7884ff45-9zq5j\" (UID: \"66a3239c-4083-424d-b164-109bb402bcfe\") " pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:38:37.681805 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.681764    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppz2r\" (UniqueName: \"kubernetes.io/projected/66a3239c-4083-424d-b164-109bb402bcfe-kube-api-access-ppz2r\") pod \"authorino-7f7884ff45-9zq5j\" (UID: \"66a3239c-4083-424d-b164-109bb402bcfe\") " pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:38:37.692215 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.692183    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 20:38:37.695041 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.695009    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f7884ff45-9zq5j"]
Nov 03 20:38:37.782821 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.782783    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66a3239c-4083-424d-b164-109bb402bcfe-tls-cert\") pod \"authorino-7f7884ff45-9zq5j\" (UID: \"66a3239c-4083-424d-b164-109bb402bcfe\") " pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:38:37.783033 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.782898    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppz2r\" (UniqueName: \"kubernetes.io/projected/66a3239c-4083-424d-b164-109bb402bcfe-kube-api-access-ppz2r\") pod \"authorino-7f7884ff45-9zq5j\" (UID: \"66a3239c-4083-424d-b164-109bb402bcfe\") " pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:38:37.785511 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.785484    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66a3239c-4083-424d-b164-109bb402bcfe-tls-cert\") pod \"authorino-7f7884ff45-9zq5j\" (UID: \"66a3239c-4083-424d-b164-109bb402bcfe\") " pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:38:37.837064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.837031    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppz2r\" (UniqueName: \"kubernetes.io/projected/66a3239c-4083-424d-b164-109bb402bcfe-kube-api-access-ppz2r\") pod \"authorino-7f7884ff45-9zq5j\" (UID: \"66a3239c-4083-424d-b164-109bb402bcfe\") " pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:38:37.852447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.852407    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ff895fdf5-np9p9"]
Nov 03 20:38:37.858998 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:38:37.858965    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf82d96b_aa8c_47ce_bda1_8f261d18c7b6.slice/crio-6f92f46d5096567c277d01ce661f24c772ff3476494350a56337f49e8086f293 WatchSource:0}: Error finding container 6f92f46d5096567c277d01ce661f24c772ff3476494350a56337f49e8086f293: Status 404 returned error can't find the container with id 6f92f46d5096567c277d01ce661f24c772ff3476494350a56337f49e8086f293
Nov 03 20:38:37.956703 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:37.956614    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:38:38.281396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:38.281292    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ff895fdf5-np9p9" event={"ID":"df82d96b-aa8c-47ce-bda1-8f261d18c7b6","Type":"ContainerStarted","Data":"6f92f46d5096567c277d01ce661f24c772ff3476494350a56337f49e8086f293"}
Nov 03 20:38:38.281396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:38.281309    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9g6sz"
Nov 03 20:38:38.340976 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:38.340934    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f7884ff45-9zq5j"]
Nov 03 20:38:38.347159 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:38:38.347128    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66a3239c_4083_424d_b164_109bb402bcfe.slice/crio-db6eb63b6d82adcb9beca6a2ad8030832ca7e25ecfe91eb12a17df5b309c55ee WatchSource:0}: Error finding container db6eb63b6d82adcb9beca6a2ad8030832ca7e25ecfe91eb12a17df5b309c55ee: Status 404 returned error can't find the container with id db6eb63b6d82adcb9beca6a2ad8030832ca7e25ecfe91eb12a17df5b309c55ee
Nov 03 20:38:38.494306 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:38.494250    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9g6sz"]
Nov 03 20:38:38.496737 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:38.496709    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9g6sz"]
Nov 03 20:38:38.818609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:38.818564    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:39.289043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:39.288955    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7884ff45-9zq5j" event={"ID":"66a3239c-4083-424d-b164-109bb402bcfe","Type":"ContainerStarted","Data":"db6eb63b6d82adcb9beca6a2ad8030832ca7e25ecfe91eb12a17df5b309c55ee"}
Nov 03 20:38:40.110447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.110408    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa678f5-63c1-42ea-bc4c-92d862ed2946" path="/var/lib/kubelet/pods/faa678f5-63c1-42ea-bc4c-92d862ed2946/volumes"
Nov 03 20:38:40.296613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.296572    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ff895fdf5-np9p9" event={"ID":"df82d96b-aa8c-47ce-bda1-8f261d18c7b6","Type":"ContainerStarted","Data":"c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67"}
Nov 03 20:38:40.306937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.296641    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5ff895fdf5-np9p9" podUID="df82d96b-aa8c-47ce-bda1-8f261d18c7b6" containerName="authorino" containerID="cri-o://c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67" gracePeriod=30
Nov 03 20:38:40.306937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.298866    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7884ff45-9zq5j" event={"ID":"66a3239c-4083-424d-b164-109bb402bcfe","Type":"ContainerStarted","Data":"49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec"}
Nov 03 20:38:40.404995 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.404939    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5ff895fdf5-np9p9" podStartSLOduration=1.610986644 podStartE2EDuration="3.404925466s" totalImagesPullingTime="1.793938822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:40.402571297 +0000 UTC m=+560.406530037" watchObservedRunningTime="2026-11-03 20:38:40.404925466 +0000 UTC m=+560.408884189"
Nov 03 20:38:40.517310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.517239    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7f7884ff45-9zq5j" podStartSLOduration=2.135883498 podStartE2EDuration="3.517217093s" totalImagesPullingTime="1.381333595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:40.513487826 +0000 UTC m=+560.517446561" watchObservedRunningTime="2026-11-03 20:38:40.517217093 +0000 UTC m=+560.521175821"
Nov 03 20:38:40.550464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.550379    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9n5jh"]
Nov 03 20:38:40.550868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.550837    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh" podUID="c7d8b932-c50b-41f2-9743-a2670c8baefb" containerName="authorino" containerID="cri-o://bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c" gracePeriod=30
Nov 03 20:38:40.569888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.569849    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:40.575991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.575963    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ff895fdf5-np9p9"
Nov 03 20:38:40.610162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.610129    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df82d96b-aa8c-47ce-bda1-8f261d18c7b6-kube-api-access-vpnsw\" (UniqueName: \"kubernetes.io/projected/df82d96b-aa8c-47ce-bda1-8f261d18c7b6-kube-api-access-vpnsw\") pod \"df82d96b-aa8c-47ce-bda1-8f261d18c7b6\" (UID: \"df82d96b-aa8c-47ce-bda1-8f261d18c7b6\") "
Nov 03 20:38:40.612448 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.612412    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df82d96b-aa8c-47ce-bda1-8f261d18c7b6-kube-api-access-vpnsw" pod "df82d96b-aa8c-47ce-bda1-8f261d18c7b6" (UID: "df82d96b-aa8c-47ce-bda1-8f261d18c7b6"). InnerVolumeSpecName "kube-api-access-vpnsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:40.711836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.711792    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpnsw\" (UniqueName: \"kubernetes.io/projected/df82d96b-aa8c-47ce-bda1-8f261d18c7b6-kube-api-access-vpnsw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:40.835006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.834974    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh"
Nov 03 20:38:40.913713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.913674    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d8b932-c50b-41f2-9743-a2670c8baefb-kube-api-access-cs6rl\" (UniqueName: \"kubernetes.io/projected/c7d8b932-c50b-41f2-9743-a2670c8baefb-kube-api-access-cs6rl\") pod \"c7d8b932-c50b-41f2-9743-a2670c8baefb\" (UID: \"c7d8b932-c50b-41f2-9743-a2670c8baefb\") "
Nov 03 20:38:40.915886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:40.915847    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d8b932-c50b-41f2-9743-a2670c8baefb-kube-api-access-cs6rl" pod "c7d8b932-c50b-41f2-9743-a2670c8baefb" (UID: "c7d8b932-c50b-41f2-9743-a2670c8baefb"). InnerVolumeSpecName "kube-api-access-cs6rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:41.015203 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.015160    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs6rl\" (UniqueName: \"kubernetes.io/projected/c7d8b932-c50b-41f2-9743-a2670c8baefb-kube-api-access-cs6rl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:41.305547 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.305463    5196 generic.go:356] "Generic (PLEG): container finished" podID="c7d8b932-c50b-41f2-9743-a2670c8baefb" containerID="bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c" exitCode=0
Nov 03 20:38:41.305547 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.305508    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh"
Nov 03 20:38:41.306045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.305544    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh" event={"ID":"c7d8b932-c50b-41f2-9743-a2670c8baefb","Type":"ContainerDied","Data":"bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c"}
Nov 03 20:38:41.306045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.305592    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9n5jh" event={"ID":"c7d8b932-c50b-41f2-9743-a2670c8baefb","Type":"ContainerDied","Data":"a508e61ddbff74a8c72ea471a51deb3d41eb0100d8a669606f4b98e806f45f9d"}
Nov 03 20:38:41.306045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.305618    5196 scope.go:122] "RemoveContainer" containerID="bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c"
Nov 03 20:38:41.308112 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.308065    5196 generic.go:356] "Generic (PLEG): container finished" podID="df82d96b-aa8c-47ce-bda1-8f261d18c7b6" containerID="c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67" exitCode=0
Nov 03 20:38:41.308220 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.308131    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ff895fdf5-np9p9"
Nov 03 20:38:41.308220 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.308133    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ff895fdf5-np9p9" event={"ID":"df82d96b-aa8c-47ce-bda1-8f261d18c7b6","Type":"ContainerDied","Data":"c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67"}
Nov 03 20:38:41.308220 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.308182    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ff895fdf5-np9p9" event={"ID":"df82d96b-aa8c-47ce-bda1-8f261d18c7b6","Type":"ContainerDied","Data":"6f92f46d5096567c277d01ce661f24c772ff3476494350a56337f49e8086f293"}
Nov 03 20:38:41.310324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.310303    5196 generic.go:356] "Generic (PLEG): container finished" podID="e67311d6-9cf7-4985-abb1-1cc9763e6156" containerID="e4c63424585f803c2fa6853f00e605df701db7da7339dd5d59aca5c83cf41371" exitCode=0
Nov 03 20:38:41.310455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.310406    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" event={"ID":"e67311d6-9cf7-4985-abb1-1cc9763e6156","Type":"ContainerDied","Data":"e4c63424585f803c2fa6853f00e605df701db7da7339dd5d59aca5c83cf41371"}
Nov 03 20:38:41.310949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.310930    5196 scope.go:122] "RemoveContainer" containerID="e4c63424585f803c2fa6853f00e605df701db7da7339dd5d59aca5c83cf41371"
Nov 03 20:38:41.312347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.312300    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:41.312457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.312367    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:41.312457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.312407    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:41.318888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.318868    5196 scope.go:122] "RemoveContainer" containerID="bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c"
Nov 03 20:38:41.319290 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:38:41.319263    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c\": container with ID starting with bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c not found: ID does not exist" containerID="bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c"
Nov 03 20:38:41.319364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.319304    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c"} err="failed to get container status \"bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c\": rpc error: code = NotFound desc = could not find container \"bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c\": container with ID starting with bcccd8b74267df0e5aad1812bd2c3f35891acf278ba3b8c18fe03de02e92812c not found: ID does not exist"
Nov 03 20:38:41.319364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.319327    5196 scope.go:122] "RemoveContainer" containerID="c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67"
Nov 03 20:38:41.328693 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.328667    5196 scope.go:122] "RemoveContainer" containerID="c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67"
Nov 03 20:38:41.329009 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:38:41.328992    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67\": container with ID starting with c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67 not found: ID does not exist" containerID="c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67"
Nov 03 20:38:41.329063 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.329018    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67"} err="failed to get container status \"c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67\": rpc error: code = NotFound desc = could not find container \"c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67\": container with ID starting with c723fef5142d065a605c5b957ac6890e82c92ea3c131362168a5a3f488f40a67 not found: ID does not exist"
Nov 03 20:38:41.461243 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.461199    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ff895fdf5-np9p9"]
Nov 03 20:38:41.464234 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.464201    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5ff895fdf5-np9p9"]
Nov 03 20:38:41.564700 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.564613    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9n5jh"]
Nov 03 20:38:41.568471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:41.568444    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9n5jh"]
Nov 03 20:38:42.108253 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:42.108221    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d8b932-c50b-41f2-9743-a2670c8baefb" path="/var/lib/kubelet/pods/c7d8b932-c50b-41f2-9743-a2670c8baefb/volumes"
Nov 03 20:38:42.108573 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:42.108560    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df82d96b-aa8c-47ce-bda1-8f261d18c7b6" path="/var/lib/kubelet/pods/df82d96b-aa8c-47ce-bda1-8f261d18c7b6/volumes"
Nov 03 20:38:42.324983 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:42.324944    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" event={"ID":"e67311d6-9cf7-4985-abb1-1cc9763e6156","Type":"ContainerStarted","Data":"7d070030ebcd57be623ee89adee0d7cc0d7774545fa03639eab453e7dcc3b8f8"}
Nov 03 20:38:42.905224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:42.905186    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:38:42.906339 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:42.906311    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rrffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:38:42.906421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:42.906379    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" podUID="e67311d6-9cf7-4985-abb1-1cc9763e6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 03 20:38:43.905084 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:43.905051    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rrffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:38:43.905530 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:43.905161    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" podUID="e67311d6-9cf7-4985-abb1-1cc9763e6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 03 20:38:44.861632 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:44.861594    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:38:44.905450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:44.905415    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rrffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:38:44.905923 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:44.905475    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" podUID="e67311d6-9cf7-4985-abb1-1cc9763e6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 03 20:38:45.905036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:45.905004    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rrffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:38:45.905233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:45.905058    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" podUID="e67311d6-9cf7-4985-abb1-1cc9763e6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 03 20:38:46.905256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:46.905215    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rrffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:38:46.905680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:46.905275    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" podUID="e67311d6-9cf7-4985-abb1-1cc9763e6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 03 20:38:47.908645 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:47.908604    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:38:47.909757 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:47.909736    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz"
Nov 03 20:38:48.483413 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:38:48.483373    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:38:57.538624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:38:57.538581    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:01.689071 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:01.689034    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:01.836302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:01.836271    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:14.982547 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:14.982501    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:15.143318 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:15.143287    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:18.485613 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:39:18.485570    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:39:20.171267 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:20.171239    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:39:20.173319 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:20.173295    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:39:20.178146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:20.178122    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:39:20.180203 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:20.180184    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:39:35.880385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.880343    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"]
Nov 03 20:39:35.880995 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.880972    5196 cpu_manager.go:414] "containerMap: removing container" podUID="df82d96b-aa8c-47ce-bda1-8f261d18c7b6" containerName="authorino"
Nov 03 20:39:35.881079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.880999    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df82d96b-aa8c-47ce-bda1-8f261d18c7b6" containerName="authorino"
Nov 03 20:39:35.881079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.881063    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c7d8b932-c50b-41f2-9743-a2670c8baefb" containerName="authorino"
Nov 03 20:39:35.881079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.881072    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d8b932-c50b-41f2-9743-a2670c8baefb" containerName="authorino"
Nov 03 20:39:35.881259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.881199    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d8b932-c50b-41f2-9743-a2670c8baefb" containerName="authorino"
Nov 03 20:39:35.881259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.881214    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="df82d96b-aa8c-47ce-bda1-8f261d18c7b6" containerName="authorino"
Nov 03 20:39:35.883748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.883727    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:35.967365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.967328    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:35.967365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.967369    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:35.967646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.967487    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822db250-6afd-4838-b96d-b64f1b7dbfc9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:35.967646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.967521    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:35.967646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.967552    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdnb4\" (UniqueName: \"kubernetes.io/projected/822db250-6afd-4838-b96d-b64f1b7dbfc9-kube-api-access-jdnb4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:35.967775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.967720    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:35.982321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.982292    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 20:39:35.982321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.982316    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mr542\""
Nov 03 20:39:35.982321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.982330    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 20:39:35.982321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.982289    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 20:39:35.984718 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.984687    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"]
Nov 03 20:39:35.984843 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:35.984687    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:36.068287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068249    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.068287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068290    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.068526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068307    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.068526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068349    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822db250-6afd-4838-b96d-b64f1b7dbfc9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.068526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068364    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.068526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068387    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdnb4\" (UniqueName: \"kubernetes.io/projected/822db250-6afd-4838-b96d-b64f1b7dbfc9-kube-api-access-jdnb4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.068782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068742    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.068848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068825    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.068895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.068855    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.070839 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.070820    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/822db250-6afd-4838-b96d-b64f1b7dbfc9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.071230 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.071209    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/822db250-6afd-4838-b96d-b64f1b7dbfc9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.125270 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.125235    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdnb4\" (UniqueName: \"kubernetes.io/projected/822db250-6afd-4838-b96d-b64f1b7dbfc9-kube-api-access-jdnb4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7nr6l\" (UID: \"822db250-6afd-4838-b96d-b64f1b7dbfc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.196379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.196281    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:36.572055 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.572020    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"]
Nov 03 20:39:36.578072 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:39:36.578047    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822db250_6afd_4838_b96d_b64f1b7dbfc9.slice/crio-e33341ea5c2f0645f56baf8768c6fd9ed8afbf41ac8807ca1b6feebdb702fc20 WatchSource:0}: Error finding container e33341ea5c2f0645f56baf8768c6fd9ed8afbf41ac8807ca1b6feebdb702fc20: Status 404 returned error can't find the container with id e33341ea5c2f0645f56baf8768c6fd9ed8afbf41ac8807ca1b6feebdb702fc20
Nov 03 20:39:36.656997 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:36.656964    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l" event={"ID":"822db250-6afd-4838-b96d-b64f1b7dbfc9","Type":"ContainerStarted","Data":"e33341ea5c2f0645f56baf8768c6fd9ed8afbf41ac8807ca1b6feebdb702fc20"}
Nov 03 20:39:39.188946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:39.188903    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:39.240904 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:39.240868    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:39:39.684263 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:39.684217    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l" event={"ID":"822db250-6afd-4838-b96d-b64f1b7dbfc9","Type":"ContainerStarted","Data":"4a41da16e9ad30f0d469b6f41dbe1a878b37095e025ecf135b3b030316cc54f8"}
Nov 03 20:39:39.793744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:39.793688    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l" podStartSLOduration=2.598195811 podStartE2EDuration="4.793672802s" totalImagesPullingTime="2.195476991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:39.791958315 +0000 UTC m=+619.795917058" watchObservedRunningTime="2026-11-03 20:39:39.793672802 +0000 UTC m=+619.797631522"
Nov 03 20:39:40.196933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:40.196895    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:44.316555 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.316523    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"]
Nov 03 20:39:44.320365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.320347    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.419513 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.419484    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 20:39:44.421671 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.421646    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"]
Nov 03 20:39:44.432967 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.432935    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:44.453339 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.453299    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fm9\" (UniqueName: \"kubernetes.io/projected/e24913d4-e48b-489c-ae78-1a091424e4fb-kube-api-access-x7fm9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.453553 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.453459    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.453621 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.453561    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.453621 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.453593    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.453700 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.453642    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24913d4-e48b-489c-ae78-1a091424e4fb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.453776 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.453749    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.554991    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24913d4-e48b-489c-ae78-1a091424e4fb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.555059    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.555080    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fm9\" (UniqueName: \"kubernetes.io/projected/e24913d4-e48b-489c-ae78-1a091424e4fb-kube-api-access-x7fm9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.555129    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.555151    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.555170    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.555649    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.555665    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.555874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.555721    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.557686 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.557658    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24913d4-e48b-489c-ae78-1a091424e4fb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.557875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.557861    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24913d4-e48b-489c-ae78-1a091424e4fb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.609843 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.609808    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fm9\" (UniqueName: \"kubernetes.io/projected/e24913d4-e48b-489c-ae78-1a091424e4fb-kube-api-access-x7fm9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k4dtv\" (UID: \"e24913d4-e48b-489c-ae78-1a091424e4fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:44.633743 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:44.633711    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:45.011016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:45.010937    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"]
Nov 03 20:39:45.017285 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:39:45.017240    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24913d4_e48b_489c_ae78_1a091424e4fb.slice/crio-3c69978696c3352e2afb109f43bdd1c0d859cf3562b23bf52e5be6e7ac9a5737 WatchSource:0}: Error finding container 3c69978696c3352e2afb109f43bdd1c0d859cf3562b23bf52e5be6e7ac9a5737: Status 404 returned error can't find the container with id 3c69978696c3352e2afb109f43bdd1c0d859cf3562b23bf52e5be6e7ac9a5737
Nov 03 20:39:45.723454 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:45.723415    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv" event={"ID":"e24913d4-e48b-489c-ae78-1a091424e4fb","Type":"ContainerStarted","Data":"d60853851ffa5d3212ccdf368cfef08cdc3c9079056a802e082fe0d3200603d6"}
Nov 03 20:39:45.723846 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:45.723489    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv" event={"ID":"e24913d4-e48b-489c-ae78-1a091424e4fb","Type":"ContainerStarted","Data":"3c69978696c3352e2afb109f43bdd1c0d859cf3562b23bf52e5be6e7ac9a5737"}
Nov 03 20:39:45.832252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:45.832190    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv" podStartSLOduration=1.214937506 podStartE2EDuration="1.832174638s" totalImagesPullingTime="617.237132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:45.830840226 +0000 UTC m=+625.834798974" watchObservedRunningTime="2026-11-03 20:39:45.832174638 +0000 UTC m=+625.836133362"
Nov 03 20:39:46.196972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:46.196935    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:46.210055 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:46.210029    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:46.431825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:46.431785    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:39:46.633937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:46.633902    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:46.749930 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:46.749892    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l"
Nov 03 20:39:48.483962 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:39:48.483925    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:39:48.583891 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:48.583861    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:52.031764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.031726    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"]
Nov 03 20:39:52.035921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.035889    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.133764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.133736    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 20:39:52.140893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.140861    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"]
Nov 03 20:39:52.141722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.141686    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:52.141922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.141892    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwp9\" (UniqueName: \"kubernetes.io/projected/42330a88-2cf2-4b6e-a877-34cbcba782d6-kube-api-access-wjwp9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.141976 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.141960    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42330a88-2cf2-4b6e-a877-34cbcba782d6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.142153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.142134    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.142234 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.142171    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.142312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.142290    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.142366 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.142344    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.244124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.244037    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.244326 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.244150    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.244326 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.244215    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwp9\" (UniqueName: \"kubernetes.io/projected/42330a88-2cf2-4b6e-a877-34cbcba782d6-kube-api-access-wjwp9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.244326 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.244265    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42330a88-2cf2-4b6e-a877-34cbcba782d6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.244532 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.244417    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.244532 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.244445    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.244965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.244939    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.245505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.245477    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.246310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.246286    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.248567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.248541    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42330a88-2cf2-4b6e-a877-34cbcba782d6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.249047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.249024    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42330a88-2cf2-4b6e-a877-34cbcba782d6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.299215 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.299131    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwp9\" (UniqueName: \"kubernetes.io/projected/42330a88-2cf2-4b6e-a877-34cbcba782d6-kube-api-access-wjwp9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t\" (UID: \"42330a88-2cf2-4b6e-a877-34cbcba782d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.352888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.352844    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:52.785406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:52.785368    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"]
Nov 03 20:39:52.792067 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:39:52.792037    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42330a88_2cf2_4b6e_a877_34cbcba782d6.slice/crio-0a90550a6822365a1991b4e91f1b60e152e95173f5f976502c532f6d9de7e75a WatchSource:0}: Error finding container 0a90550a6822365a1991b4e91f1b60e152e95173f5f976502c532f6d9de7e75a: Status 404 returned error can't find the container with id 0a90550a6822365a1991b4e91f1b60e152e95173f5f976502c532f6d9de7e75a
Nov 03 20:39:53.779151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:53.779035    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t" event={"ID":"42330a88-2cf2-4b6e-a877-34cbcba782d6","Type":"ContainerStarted","Data":"42aa6265fe157c0bcdb247e972d49ee1d48e72b3f4bcc1824c673cbdbc9c241f"}
Nov 03 20:39:53.779151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:53.779146    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t" event={"ID":"42330a88-2cf2-4b6e-a877-34cbcba782d6","Type":"ContainerStarted","Data":"0a90550a6822365a1991b4e91f1b60e152e95173f5f976502c532f6d9de7e75a"}
Nov 03 20:39:53.888325 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:53.888270    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t" podStartSLOduration=2.28640945 podStartE2EDuration="2.888257704s" totalImagesPullingTime="601.848254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:53.887804919 +0000 UTC m=+633.891763660" watchObservedRunningTime="2026-11-03 20:39:53.888257704 +0000 UTC m=+633.892216427"
Nov 03 20:39:54.354039 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:54.353991    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:39:54.634750 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:54.634649    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:54.647439 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:54.647409    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:54.795696 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:54.795664    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k4dtv"
Nov 03 20:39:55.516423 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:55.516388    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:39:56.040334 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:56.040285    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:59.398260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.398223    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"]
Nov 03 20:39:59.402140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.402086    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.504171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.504130    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 20:39:59.506658 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.506634    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"]
Nov 03 20:39:59.516364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.516335    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178b98c9-47eb-4858-a069-469155095085-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.516606 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.516583    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.516877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.516835    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.517142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.517111    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pxj6\" (UniqueName: \"kubernetes.io/projected/178b98c9-47eb-4858-a069-469155095085-kube-api-access-4pxj6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.517233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.517195    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.517313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.517237    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.517760 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.517727    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:59.618807 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.618768    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178b98c9-47eb-4858-a069-469155095085-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.619017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.618824    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.619017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.618846    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.619017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.618870    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pxj6\" (UniqueName: \"kubernetes.io/projected/178b98c9-47eb-4858-a069-469155095085-kube-api-access-4pxj6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.619017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.618903    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.619017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.618921    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.619425 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.619392    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.619555 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.619427    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.619555 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.619411    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.621451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.621428    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178b98c9-47eb-4858-a069-469155095085-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.621629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.621611    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178b98c9-47eb-4858-a069-469155095085-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.673118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.673012    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pxj6\" (UniqueName: \"kubernetes.io/projected/178b98c9-47eb-4858-a069-469155095085-kube-api-access-4pxj6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m\" (UID: \"178b98c9-47eb-4858-a069-469155095085\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:39:59.716075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:39:59.716040    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:40:00.095791 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:00.095746    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"]
Nov 03 20:40:00.102605 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:40:00.102571    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178b98c9_47eb_4858_a069_469155095085.slice/crio-e6efdb4f49522be19ee425a64021474a1f8a1bcdfe4804c0edbfe14e2672660c WatchSource:0}: Error finding container e6efdb4f49522be19ee425a64021474a1f8a1bcdfe4804c0edbfe14e2672660c: Status 404 returned error can't find the container with id e6efdb4f49522be19ee425a64021474a1f8a1bcdfe4804c0edbfe14e2672660c
Nov 03 20:40:00.104050 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:00.104027    5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:40:00.823059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:00.823030    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m" event={"ID":"178b98c9-47eb-4858-a069-469155095085","Type":"ContainerStarted","Data":"e6efdb4f49522be19ee425a64021474a1f8a1bcdfe4804c0edbfe14e2672660c"}
Nov 03 20:40:01.831335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:01.831299    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m" event={"ID":"178b98c9-47eb-4858-a069-469155095085","Type":"ContainerStarted","Data":"09bf1daa48f1c78069aea25bd419c2104d3d9205f82e6fb4ccc31699bfda4ce3"}
Nov 03 20:40:01.957544 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:01.957488    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m" podStartSLOduration=2.265402583 podStartE2EDuration="2.957475067s" totalImagesPullingTime="692.072484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:01.955804334 +0000 UTC m=+641.959763058" watchObservedRunningTime="2026-11-03 20:40:01.957475067 +0000 UTC m=+641.961433791"
Nov 03 20:40:02.353695 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.353658    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:40:02.366425 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.366399    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:40:02.706768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.706679    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"]
Nov 03 20:40:02.711428 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.711400    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.717154 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.717125    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:40:02.750356 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.750320    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.750522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.750401    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.750562 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.750527    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.750604 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.750573    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxbg\" (UniqueName: \"kubernetes.io/projected/ed185af4-9540-4841-a9c3-bd14cf61e7e2-kube-api-access-qjxbg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.750676 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.750658    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed185af4-9540-4841-a9c3-bd14cf61e7e2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.750711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.750699    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.762615 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.762584    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 20:40:02.765390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.765361    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"]
Nov 03 20:40:02.828277 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.828236    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:02.852347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.852314    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t"
Nov 03 20:40:02.855193 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.855162    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed185af4-9540-4841-a9c3-bd14cf61e7e2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.855458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.855430    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.855600 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.855539    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.855668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.855612    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.855728 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.855668    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.855728 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.855699    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxbg\" (UniqueName: \"kubernetes.io/projected/ed185af4-9540-4841-a9c3-bd14cf61e7e2-kube-api-access-qjxbg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.855884 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.855863    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.856171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.856148    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.856380 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.856355    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.858891 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.858868    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed185af4-9540-4841-a9c3-bd14cf61e7e2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.859003 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.858980    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed185af4-9540-4841-a9c3-bd14cf61e7e2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:02.929846 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:02.929795    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxbg\" (UniqueName: \"kubernetes.io/projected/ed185af4-9540-4841-a9c3-bd14cf61e7e2-kube-api-access-qjxbg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz\" (UID: \"ed185af4-9540-4841-a9c3-bd14cf61e7e2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:03.025210 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:03.024130    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:03.416821 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:03.416786    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"]
Nov 03 20:40:03.423951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:40:03.423923    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded185af4_9540_4841_a9c3_bd14cf61e7e2.slice/crio-65a765880e7226907f58b0c67101d1e9b24e6b08d85a15b0091c3984c5fc75e4 WatchSource:0}: Error finding container 65a765880e7226907f58b0c67101d1e9b24e6b08d85a15b0091c3984c5fc75e4: Status 404 returned error can't find the container with id 65a765880e7226907f58b0c67101d1e9b24e6b08d85a15b0091c3984c5fc75e4
Nov 03 20:40:03.850365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:03.850325    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" event={"ID":"ed185af4-9540-4841-a9c3-bd14cf61e7e2","Type":"ContainerStarted","Data":"65a765880e7226907f58b0c67101d1e9b24e6b08d85a15b0091c3984c5fc75e4"}
Nov 03 20:40:04.857746 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:04.857708    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" event={"ID":"ed185af4-9540-4841-a9c3-bd14cf61e7e2","Type":"ContainerStarted","Data":"4c7368c6b265692ab9749589055baa3eb6dbd6869237291362e7c3b4b5cd7928"}
Nov 03 20:40:04.968672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:04.968616    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podStartSLOduration=2.366937697 podStartE2EDuration="2.968601563s" totalImagesPullingTime="601.663866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:04.965825505 +0000 UTC m=+644.969784249" watchObservedRunningTime="2026-11-03 20:40:04.968601563 +0000 UTC m=+644.972560287"
Nov 03 20:40:05.025001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:05.024960    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:05.645357 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:05.645309    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:09.553617 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:09.553574    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:40:09.716404 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:09.716363    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:40:09.729843 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:09.729811    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:40:09.902732 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:09.902688    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m"
Nov 03 20:40:13.024526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:13.024490    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:13.037649 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:13.037620    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:13.923836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:13.923801    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:40:17.981785 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:17.981741    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"]
Nov 03 20:40:17.987615 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:17.987586    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.040563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.040512    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"]
Nov 03 20:40:18.127018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.126977    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.127228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.127025    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.127228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.127181    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.127228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.127219    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.127363 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.127240    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.127363 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.127264    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqf2\" (UniqueName: \"kubernetes.io/projected/1cba3ff6-170a-45dc-8cf7-1517813237c9-kube-api-access-2xqf2\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.127431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.127360    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.127431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.127413    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cba3ff6-170a-45dc-8cf7-1517813237c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.127521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.127466    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228123    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228171    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228316    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqf2\" (UniqueName: \"kubernetes.io/projected/1cba3ff6-170a-45dc-8cf7-1517813237c9-kube-api-access-2xqf2\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228396    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228471    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cba3ff6-170a-45dc-8cf7-1517813237c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228532    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228603    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228639    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.228811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.228786    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.229079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.229050    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.229270 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.229243    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.229343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.229274    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.229494 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.229464    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.229852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.229826    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cba3ff6-170a-45dc-8cf7-1517813237c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.230832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.230808    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.231049 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.231030    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.283460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.283383    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cba3ff6-170a-45dc-8cf7-1517813237c9-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.283626 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.283569    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqf2\" (UniqueName: \"kubernetes.io/projected/1cba3ff6-170a-45dc-8cf7-1517813237c9-kube-api-access-2xqf2\") pod \"maas-default-gateway-openshift-default-696f47c6c-fm4zc\" (UID: \"1cba3ff6-170a-45dc-8cf7-1517813237c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.302313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.302285    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:18.484701 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:40:18.484661    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:40:18.728244 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.728211    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"]
Nov 03 20:40:18.734310 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:40:18.734278    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cba3ff6_170a_45dc_8cf7_1517813237c9.slice/crio-95c04cd87da773dc8e34194cf22cf9efc7a04cc85e365ed350dae23a91aa872b WatchSource:0}: Error finding container 95c04cd87da773dc8e34194cf22cf9efc7a04cc85e365ed350dae23a91aa872b: Status 404 returned error can't find the container with id 95c04cd87da773dc8e34194cf22cf9efc7a04cc85e365ed350dae23a91aa872b
Nov 03 20:40:18.736726 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.736688    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:40:18.736799 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.736773    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:40:18.736843 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.736807    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:40:18.944901 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.944864    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" event={"ID":"1cba3ff6-170a-45dc-8cf7-1517813237c9","Type":"ContainerStarted","Data":"710006f54a5a03c8604e0c08c085fa1dc7edc1ccf7224c1eec5b806e7af5eccf"}
Nov 03 20:40:18.945078 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:18.944923    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" event={"ID":"1cba3ff6-170a-45dc-8cf7-1517813237c9","Type":"ContainerStarted","Data":"95c04cd87da773dc8e34194cf22cf9efc7a04cc85e365ed350dae23a91aa872b"}
Nov 03 20:40:19.057593 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:19.057487    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podStartSLOduration=2.057473729 podStartE2EDuration="2.057473729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:19.055328416 +0000 UTC m=+659.059287159" watchObservedRunningTime="2026-11-03 20:40:19.057473729 +0000 UTC m=+659.061432453"
Nov 03 20:40:19.302853 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:19.302821    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:19.304318 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:19.304290    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:40:19.304451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:19.304341    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 20:40:20.303034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:20.302995    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:40:20.303493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:20.303052    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 20:40:21.302931 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:21.302890    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:40:21.303143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:21.302944    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 20:40:22.303344 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:22.303306    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:40:22.303752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:22.303365    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 20:40:23.302775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:23.302737    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:40:23.303003 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:23.302806    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 20:40:24.303406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:24.303367    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:40:24.303833 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:24.303426    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 20:40:25.302879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:25.302844    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:40:25.303072 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:25.302911    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 03 20:40:27.009886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.009852    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:27.010371 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.010240    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:27.011169 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.011151    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc"
Nov 03 20:40:27.535013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.534978    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7684488598-v9cnc"]
Nov 03 20:40:27.539348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.539328    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7684488598-v9cnc"
Nov 03 20:40:27.590418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.590385    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7684488598-v9cnc"]
Nov 03 20:40:27.634334 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.634298    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/795cbb83-ddae-4f68-a84e-aad5d3a02340-tls-cert\") pod \"authorino-7684488598-v9cnc\" (UID: \"795cbb83-ddae-4f68-a84e-aad5d3a02340\") " pod="kuadrant-system/authorino-7684488598-v9cnc"
Nov 03 20:40:27.634521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.634366    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx8c4\" (UniqueName: \"kubernetes.io/projected/795cbb83-ddae-4f68-a84e-aad5d3a02340-kube-api-access-tx8c4\") pod \"authorino-7684488598-v9cnc\" (UID: \"795cbb83-ddae-4f68-a84e-aad5d3a02340\") " pod="kuadrant-system/authorino-7684488598-v9cnc"
Nov 03 20:40:27.735810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.735779    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/795cbb83-ddae-4f68-a84e-aad5d3a02340-tls-cert\") pod \"authorino-7684488598-v9cnc\" (UID: \"795cbb83-ddae-4f68-a84e-aad5d3a02340\") " pod="kuadrant-system/authorino-7684488598-v9cnc"
Nov 03 20:40:27.736014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.735849    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx8c4\" (UniqueName: \"kubernetes.io/projected/795cbb83-ddae-4f68-a84e-aad5d3a02340-kube-api-access-tx8c4\") pod \"authorino-7684488598-v9cnc\" (UID: \"795cbb83-ddae-4f68-a84e-aad5d3a02340\") " pod="kuadrant-system/authorino-7684488598-v9cnc"
Nov 03 20:40:27.738724 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.738696    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/795cbb83-ddae-4f68-a84e-aad5d3a02340-tls-cert\") pod \"authorino-7684488598-v9cnc\" (UID: \"795cbb83-ddae-4f68-a84e-aad5d3a02340\") " pod="kuadrant-system/authorino-7684488598-v9cnc"
Nov 03 20:40:27.789925 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.789857    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx8c4\" (UniqueName: \"kubernetes.io/projected/795cbb83-ddae-4f68-a84e-aad5d3a02340-kube-api-access-tx8c4\") pod \"authorino-7684488598-v9cnc\" (UID: \"795cbb83-ddae-4f68-a84e-aad5d3a02340\") " pod="kuadrant-system/authorino-7684488598-v9cnc"
Nov 03 20:40:27.851538 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:27.851511    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7684488598-v9cnc"
Nov 03 20:40:28.222066 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:28.222024    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7684488598-v9cnc"]
Nov 03 20:40:28.228121 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:40:28.228073    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod795cbb83_ddae_4f68_a84e_aad5d3a02340.slice/crio-fdd97162126283f773f6f35d4e1f7c3d5370e26654ccb84197cc7c41c50e56c8 WatchSource:0}: Error finding container fdd97162126283f773f6f35d4e1f7c3d5370e26654ccb84197cc7c41c50e56c8: Status 404 returned error can't find the container with id fdd97162126283f773f6f35d4e1f7c3d5370e26654ccb84197cc7c41c50e56c8
Nov 03 20:40:29.016718 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:29.016681    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7684488598-v9cnc" event={"ID":"795cbb83-ddae-4f68-a84e-aad5d3a02340","Type":"ContainerStarted","Data":"fdd97162126283f773f6f35d4e1f7c3d5370e26654ccb84197cc7c41c50e56c8"}
Nov 03 20:40:30.024711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.024662    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7684488598-v9cnc" event={"ID":"795cbb83-ddae-4f68-a84e-aad5d3a02340","Type":"ContainerStarted","Data":"d41b151a0cef1b6565bd78eefe8b5b4afc6c0d09d0d74c216b8933ca01f0eb51"}
Nov 03 20:40:30.133638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.133527    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7684488598-v9cnc" podStartSLOduration=1.4953593760000001 podStartE2EDuration="3.133514185s" totalImagesPullingTime="1.638154809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:30.131942414 +0000 UTC m=+670.135901139" watchObservedRunningTime="2026-11-03 20:40:30.133514185 +0000 UTC m=+670.137472908"
Nov 03 20:40:30.157440 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.157398    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f7884ff45-9zq5j"]
Nov 03 20:40:30.157711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.157661    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7f7884ff45-9zq5j" podUID="66a3239c-4083-424d-b164-109bb402bcfe" containerName="authorino" containerID="cri-o://49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec" gracePeriod=30
Nov 03 20:40:30.420240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.420211    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:40:30.466654 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.466622    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a3239c-4083-424d-b164-109bb402bcfe-tls-cert\" (UniqueName: \"kubernetes.io/secret/66a3239c-4083-424d-b164-109bb402bcfe-tls-cert\") pod \"66a3239c-4083-424d-b164-109bb402bcfe\" (UID: \"66a3239c-4083-424d-b164-109bb402bcfe\") "
Nov 03 20:40:30.466861 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.466843    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a3239c-4083-424d-b164-109bb402bcfe-kube-api-access-ppz2r\" (UniqueName: \"kubernetes.io/projected/66a3239c-4083-424d-b164-109bb402bcfe-kube-api-access-ppz2r\") pod \"66a3239c-4083-424d-b164-109bb402bcfe\" (UID: \"66a3239c-4083-424d-b164-109bb402bcfe\") "
Nov 03 20:40:30.469259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.469226    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a3239c-4083-424d-b164-109bb402bcfe-kube-api-access-ppz2r" pod "66a3239c-4083-424d-b164-109bb402bcfe" (UID: "66a3239c-4083-424d-b164-109bb402bcfe"). InnerVolumeSpecName "kube-api-access-ppz2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:30.481760 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.481716    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a3239c-4083-424d-b164-109bb402bcfe-tls-cert" pod "66a3239c-4083-424d-b164-109bb402bcfe" (UID: "66a3239c-4083-424d-b164-109bb402bcfe"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:30.568162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.568126    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppz2r\" (UniqueName: \"kubernetes.io/projected/66a3239c-4083-424d-b164-109bb402bcfe-kube-api-access-ppz2r\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:30.568162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:30.568161    5196 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66a3239c-4083-424d-b164-109bb402bcfe-tls-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:31.032260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.032229    5196 generic.go:356] "Generic (PLEG): container finished" podID="66a3239c-4083-424d-b164-109bb402bcfe" containerID="49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec" exitCode=0
Nov 03 20:40:31.032683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.032285    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7884ff45-9zq5j"
Nov 03 20:40:31.032683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.032318    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7884ff45-9zq5j" event={"ID":"66a3239c-4083-424d-b164-109bb402bcfe","Type":"ContainerDied","Data":"49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec"}
Nov 03 20:40:31.032683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.032362    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7884ff45-9zq5j" event={"ID":"66a3239c-4083-424d-b164-109bb402bcfe","Type":"ContainerDied","Data":"db6eb63b6d82adcb9beca6a2ad8030832ca7e25ecfe91eb12a17df5b309c55ee"}
Nov 03 20:40:31.032683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.032385    5196 scope.go:122] "RemoveContainer" containerID="49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec"
Nov 03 20:40:31.043557 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.043536    5196 scope.go:122] "RemoveContainer" containerID="49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec"
Nov 03 20:40:31.043881 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:40:31.043857    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec\": container with ID starting with 49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec not found: ID does not exist" containerID="49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec"
Nov 03 20:40:31.043935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.043892    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec"} err="failed to get container status \"49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec\": rpc error: code = NotFound desc = could not find container \"49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec\": container with ID starting with 49b88326ca6eb4239dd933d0f0b9e73f208cbb521de61b9e3709b2075957dcec not found: ID does not exist"
Nov 03 20:40:31.188060 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.187976    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f7884ff45-9zq5j"]
Nov 03 20:40:31.191522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:31.191491    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f7884ff45-9zq5j"]
Nov 03 20:40:32.115205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:32.115171    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a3239c-4083-424d-b164-109bb402bcfe" path="/var/lib/kubelet/pods/66a3239c-4083-424d-b164-109bb402bcfe/volumes"
Nov 03 20:40:37.681618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.681582    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-769788cd69-6rkhf"]
Nov 03 20:40:37.682182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.682165    5196 cpu_manager.go:414] "containerMap: removing container" podUID="66a3239c-4083-424d-b164-109bb402bcfe" containerName="authorino"
Nov 03 20:40:37.682250 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.682185    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a3239c-4083-424d-b164-109bb402bcfe" containerName="authorino"
Nov 03 20:40:37.682306 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.682291    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a3239c-4083-424d-b164-109bb402bcfe" containerName="authorino"
Nov 03 20:40:37.687234 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.687213    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:37.696885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.696855    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:37.739830 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.739789    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-769788cd69-6rkhf"]
Nov 03 20:40:37.770622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.770587    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fb48d2-55e2-4eaa-8874-510e432ecc83-metrics-certs\") pod \"ai-gateway-controller-769788cd69-6rkhf\" (UID: \"52fb48d2-55e2-4eaa-8874-510e432ecc83\") " pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:37.770622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.770627    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhp5\" (UniqueName: \"kubernetes.io/projected/52fb48d2-55e2-4eaa-8874-510e432ecc83-kube-api-access-4mhp5\") pod \"ai-gateway-controller-769788cd69-6rkhf\" (UID: \"52fb48d2-55e2-4eaa-8874-510e432ecc83\") " pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:37.783256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.783222    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 20:40:37.783566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.783524    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 20:40:37.783803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.783588    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-dl7f5\""
Nov 03 20:40:37.848839 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.848796    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:37.871612 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.871567    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fb48d2-55e2-4eaa-8874-510e432ecc83-metrics-certs\") pod \"ai-gateway-controller-769788cd69-6rkhf\" (UID: \"52fb48d2-55e2-4eaa-8874-510e432ecc83\") " pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:37.871803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.871617    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhp5\" (UniqueName: \"kubernetes.io/projected/52fb48d2-55e2-4eaa-8874-510e432ecc83-kube-api-access-4mhp5\") pod \"ai-gateway-controller-769788cd69-6rkhf\" (UID: \"52fb48d2-55e2-4eaa-8874-510e432ecc83\") " pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:37.874298 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.874269    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fb48d2-55e2-4eaa-8874-510e432ecc83-metrics-certs\") pod \"ai-gateway-controller-769788cd69-6rkhf\" (UID: \"52fb48d2-55e2-4eaa-8874-510e432ecc83\") " pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:37.925957 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:37.925925    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhp5\" (UniqueName: \"kubernetes.io/projected/52fb48d2-55e2-4eaa-8874-510e432ecc83-kube-api-access-4mhp5\") pod \"ai-gateway-controller-769788cd69-6rkhf\" (UID: \"52fb48d2-55e2-4eaa-8874-510e432ecc83\") " pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:38.001048 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:38.000954    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:38.372159 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:38.372124    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-769788cd69-6rkhf"]
Nov 03 20:40:38.378425 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:40:38.378388    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fb48d2_55e2_4eaa_8874_510e432ecc83.slice/crio-175ef591caa51d4383eaa4d63c3d91617a0bc95da9ef2e92487739a1486a4af1 WatchSource:0}: Error finding container 175ef591caa51d4383eaa4d63c3d91617a0bc95da9ef2e92487739a1486a4af1: Status 404 returned error can't find the container with id 175ef591caa51d4383eaa4d63c3d91617a0bc95da9ef2e92487739a1486a4af1
Nov 03 20:40:39.088511 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:39.088469    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf" event={"ID":"52fb48d2-55e2-4eaa-8874-510e432ecc83","Type":"ContainerStarted","Data":"175ef591caa51d4383eaa4d63c3d91617a0bc95da9ef2e92487739a1486a4af1"}
Nov 03 20:40:39.457409 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:39.457275    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:46.133992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:46.133954    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf" event={"ID":"52fb48d2-55e2-4eaa-8874-510e432ecc83","Type":"ContainerStarted","Data":"43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf"}
Nov 03 20:40:46.134426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:46.134054    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:46.245014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:46.244962    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf" podStartSLOduration=2.4781729439999998 podStartE2EDuration="9.244943474s" totalImagesPullingTime="6.76677053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:46.243274326 +0000 UTC m=+686.247233063" watchObservedRunningTime="2026-11-03 20:40:46.244943474 +0000 UTC m=+686.248902200"
Nov 03 20:40:47.884052 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:47.884017    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:48.074317 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.074283    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:48.277571 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.277476    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"]
Nov 03 20:40:48.281622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.281597    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:48.335499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.335466    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"]
Nov 03 20:40:48.378454 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.378421    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 20:40:48.378662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.378466    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vjh9s\""
Nov 03 20:40:48.382421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.382388    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39296fa-08ee-4504-b9db-ee97ab3e7d4b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7pv4c\" (UID: \"b39296fa-08ee-4504-b9db-ee97ab3e7d4b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:48.382574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.382456    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7pm\" (UniqueName: \"kubernetes.io/projected/b39296fa-08ee-4504-b9db-ee97ab3e7d4b-kube-api-access-cf7pm\") pod \"payload-pre-processing-7b98bb678c-7pv4c\" (UID: \"b39296fa-08ee-4504-b9db-ee97ab3e7d4b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:48.428693 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.428641    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-b785fc555-gj4ww"]
Nov 03 20:40:48.433799 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.433765    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:48.477596 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.477563    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-b785fc555-gj4ww"]
Nov 03 20:40:48.483376 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.483327    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39296fa-08ee-4504-b9db-ee97ab3e7d4b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7pv4c\" (UID: \"b39296fa-08ee-4504-b9db-ee97ab3e7d4b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:48.483565 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.483416    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7pm\" (UniqueName: \"kubernetes.io/projected/b39296fa-08ee-4504-b9db-ee97ab3e7d4b-kube-api-access-cf7pm\") pod \"payload-pre-processing-7b98bb678c-7pv4c\" (UID: \"b39296fa-08ee-4504-b9db-ee97ab3e7d4b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:48.484084 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.484060    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39296fa-08ee-4504-b9db-ee97ab3e7d4b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7pv4c\" (UID: \"b39296fa-08ee-4504-b9db-ee97ab3e7d4b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:48.492906 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:40:48.492867    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:40:48.537524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.537447    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7pm\" (UniqueName: \"kubernetes.io/projected/b39296fa-08ee-4504-b9db-ee97ab3e7d4b-kube-api-access-cf7pm\") pod \"payload-pre-processing-7b98bb678c-7pv4c\" (UID: \"b39296fa-08ee-4504-b9db-ee97ab3e7d4b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:48.584194 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.584147    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d289e2d2-7ef4-4218-80db-79ef4ea15d0e-plugins-config-volume\") pod \"payload-processing-b785fc555-gj4ww\" (UID: \"d289e2d2-7ef4-4218-80db-79ef4ea15d0e\") " pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:48.584372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.584262    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz69z\" (UniqueName: \"kubernetes.io/projected/d289e2d2-7ef4-4218-80db-79ef4ea15d0e-kube-api-access-lz69z\") pod \"payload-processing-b785fc555-gj4ww\" (UID: \"d289e2d2-7ef4-4218-80db-79ef4ea15d0e\") " pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:48.595870 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.595837    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:48.685162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.685128    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz69z\" (UniqueName: \"kubernetes.io/projected/d289e2d2-7ef4-4218-80db-79ef4ea15d0e-kube-api-access-lz69z\") pod \"payload-processing-b785fc555-gj4ww\" (UID: \"d289e2d2-7ef4-4218-80db-79ef4ea15d0e\") " pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:48.685368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.685307    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d289e2d2-7ef4-4218-80db-79ef4ea15d0e-plugins-config-volume\") pod \"payload-processing-b785fc555-gj4ww\" (UID: \"d289e2d2-7ef4-4218-80db-79ef4ea15d0e\") " pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:48.686112 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.686063    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d289e2d2-7ef4-4218-80db-79ef4ea15d0e-plugins-config-volume\") pod \"payload-processing-b785fc555-gj4ww\" (UID: \"d289e2d2-7ef4-4218-80db-79ef4ea15d0e\") " pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:48.739419 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.739388    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz69z\" (UniqueName: \"kubernetes.io/projected/d289e2d2-7ef4-4218-80db-79ef4ea15d0e-kube-api-access-lz69z\") pod \"payload-processing-b785fc555-gj4ww\" (UID: \"d289e2d2-7ef4-4218-80db-79ef4ea15d0e\") " pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:48.749594 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.749556    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:48.979288 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:48.979252    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"]
Nov 03 20:40:48.984715 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:40:48.984684    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39296fa_08ee_4504_b9db_ee97ab3e7d4b.slice/crio-cf275cb91e687c42ca73f2b6afc1e8eda56805f68dc108d59c957ee18eafee96 WatchSource:0}: Error finding container cf275cb91e687c42ca73f2b6afc1e8eda56805f68dc108d59c957ee18eafee96: Status 404 returned error can't find the container with id cf275cb91e687c42ca73f2b6afc1e8eda56805f68dc108d59c957ee18eafee96
Nov 03 20:40:49.139540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:49.136308    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-b785fc555-gj4ww"]
Nov 03 20:40:49.149574 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:40:49.149532    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd289e2d2_7ef4_4218_80db_79ef4ea15d0e.slice/crio-2acad1fa255b139c294997ee074cfe0aa9816aa945ab2e1532cde9f648a2dab5 WatchSource:0}: Error finding container 2acad1fa255b139c294997ee074cfe0aa9816aa945ab2e1532cde9f648a2dab5: Status 404 returned error can't find the container with id 2acad1fa255b139c294997ee074cfe0aa9816aa945ab2e1532cde9f648a2dab5
Nov 03 20:40:49.157368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:49.157337    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c" event={"ID":"b39296fa-08ee-4504-b9db-ee97ab3e7d4b","Type":"ContainerStarted","Data":"cf275cb91e687c42ca73f2b6afc1e8eda56805f68dc108d59c957ee18eafee96"}
Nov 03 20:40:50.170006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:50.169916    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b785fc555-gj4ww" event={"ID":"d289e2d2-7ef4-4218-80db-79ef4ea15d0e","Type":"ContainerStarted","Data":"2acad1fa255b139c294997ee074cfe0aa9816aa945ab2e1532cde9f648a2dab5"}
Nov 03 20:40:52.186374 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:52.186336    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b785fc555-gj4ww" event={"ID":"d289e2d2-7ef4-4218-80db-79ef4ea15d0e","Type":"ContainerStarted","Data":"a743b1df8d3c4d223993397875335fe2af98d18d3f3d5ba7df56f5519d2fe83f"}
Nov 03 20:40:52.186818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:52.186439    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:40:52.188020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:52.187991    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c" event={"ID":"b39296fa-08ee-4504-b9db-ee97ab3e7d4b","Type":"ContainerStarted","Data":"64461a2ff3b7155d66be916a78fcf999a8fbe82f0e61f45a86b1ea9e5a70bdc2"}
Nov 03 20:40:52.188172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:52.188144    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:40:52.296566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:52.296513    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-b785fc555-gj4ww" podStartSLOduration=1.391277053 podStartE2EDuration="4.296497852s" totalImagesPullingTime="2.905220799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:52.294900596 +0000 UTC m=+692.298859342" watchObservedRunningTime="2026-11-03 20:40:52.296497852 +0000 UTC m=+692.300456615"
Nov 03 20:40:57.143587 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.143560    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:40:57.256282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.256226    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c" podStartSLOduration=6.150616556 podStartE2EDuration="9.256211825s" totalImagesPullingTime="3.105595269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:52.403748342 +0000 UTC m=+692.407707068" watchObservedRunningTime="2026-11-03 20:40:57.256211825 +0000 UTC m=+697.260170548"
Nov 03 20:40:57.473857 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.473771    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-856cb7577-6mcxx"]
Nov 03 20:40:57.477568 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.477546    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:57.530458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.530413    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-856cb7577-6mcxx"]
Nov 03 20:40:57.576470 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.576429    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckkw4\" (UniqueName: \"kubernetes.io/projected/192e8de2-7248-4008-bf02-1873cad17c6a-kube-api-access-ckkw4\") pod \"ai-gateway-controller-856cb7577-6mcxx\" (UID: \"192e8de2-7248-4008-bf02-1873cad17c6a\") " pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:57.576652 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.576487    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/192e8de2-7248-4008-bf02-1873cad17c6a-metrics-certs\") pod \"ai-gateway-controller-856cb7577-6mcxx\" (UID: \"192e8de2-7248-4008-bf02-1873cad17c6a\") " pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:57.677047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.677008    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckkw4\" (UniqueName: \"kubernetes.io/projected/192e8de2-7248-4008-bf02-1873cad17c6a-kube-api-access-ckkw4\") pod \"ai-gateway-controller-856cb7577-6mcxx\" (UID: \"192e8de2-7248-4008-bf02-1873cad17c6a\") " pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:57.677288 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.677052    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/192e8de2-7248-4008-bf02-1873cad17c6a-metrics-certs\") pod \"ai-gateway-controller-856cb7577-6mcxx\" (UID: \"192e8de2-7248-4008-bf02-1873cad17c6a\") " pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:57.679720 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.679692    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/192e8de2-7248-4008-bf02-1873cad17c6a-metrics-certs\") pod \"ai-gateway-controller-856cb7577-6mcxx\" (UID: \"192e8de2-7248-4008-bf02-1873cad17c6a\") " pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:57.730977 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.730909    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckkw4\" (UniqueName: \"kubernetes.io/projected/192e8de2-7248-4008-bf02-1873cad17c6a-kube-api-access-ckkw4\") pod \"ai-gateway-controller-856cb7577-6mcxx\" (UID: \"192e8de2-7248-4008-bf02-1873cad17c6a\") " pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:57.791880 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:57.791847    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:58.162538 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:58.162503    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-856cb7577-6mcxx"]
Nov 03 20:40:58.168547 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:40:58.168512    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192e8de2_7248_4008_bf02_1873cad17c6a.slice/crio-91b9092af272ea9d985563c64e3298bf5a1b845c5e87ccafa70465a81f8676fe WatchSource:0}: Error finding container 91b9092af272ea9d985563c64e3298bf5a1b845c5e87ccafa70465a81f8676fe: Status 404 returned error can't find the container with id 91b9092af272ea9d985563c64e3298bf5a1b845c5e87ccafa70465a81f8676fe
Nov 03 20:40:58.225837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:58.225799    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx" event={"ID":"192e8de2-7248-4008-bf02-1873cad17c6a","Type":"ContainerStarted","Data":"91b9092af272ea9d985563c64e3298bf5a1b845c5e87ccafa70465a81f8676fe"}
Nov 03 20:40:59.234054 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:59.234021    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx" event={"ID":"192e8de2-7248-4008-bf02-1873cad17c6a","Type":"ContainerStarted","Data":"64fed706180ca658c0ee62e32358a3d0a70b101a0cd64c506dd986b8e2323e12"}
Nov 03 20:40:59.234456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:59.234085    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:40:59.343029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:40:59.342969    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx" podStartSLOduration=1.516102182 podStartE2EDuration="2.342952249s" totalImagesPullingTime="826.850067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:59.340682077 +0000 UTC m=+699.344640819" watchObservedRunningTime="2026-11-03 20:40:59.342952249 +0000 UTC m=+699.346910972"
Nov 03 20:41:03.197640 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:03.197610    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7pv4c"
Nov 03 20:41:03.198058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:03.197712    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-b785fc555-gj4ww"
Nov 03 20:41:10.242686 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.242654    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-856cb7577-6mcxx"
Nov 03 20:41:10.400965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.400931    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-769788cd69-6rkhf"]
Nov 03 20:41:10.401311 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.401210    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf" podUID="52fb48d2-55e2-4eaa-8874-510e432ecc83" containerName="manager" containerID="cri-o://43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf" gracePeriod=30
Nov 03 20:41:10.660256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.660231    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:41:10.704763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.704726    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52fb48d2-55e2-4eaa-8874-510e432ecc83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fb48d2-55e2-4eaa-8874-510e432ecc83-metrics-certs\") pod \"52fb48d2-55e2-4eaa-8874-510e432ecc83\" (UID: \"52fb48d2-55e2-4eaa-8874-510e432ecc83\") "
Nov 03 20:41:10.704926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.704866    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fb48d2-55e2-4eaa-8874-510e432ecc83-kube-api-access-4mhp5\" (UniqueName: \"kubernetes.io/projected/52fb48d2-55e2-4eaa-8874-510e432ecc83-kube-api-access-4mhp5\") pod \"52fb48d2-55e2-4eaa-8874-510e432ecc83\" (UID: \"52fb48d2-55e2-4eaa-8874-510e432ecc83\") "
Nov 03 20:41:10.706937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.706898    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fb48d2-55e2-4eaa-8874-510e432ecc83-metrics-certs" pod "52fb48d2-55e2-4eaa-8874-510e432ecc83" (UID: "52fb48d2-55e2-4eaa-8874-510e432ecc83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:41:10.706937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.706919    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fb48d2-55e2-4eaa-8874-510e432ecc83-kube-api-access-4mhp5" pod "52fb48d2-55e2-4eaa-8874-510e432ecc83" (UID: "52fb48d2-55e2-4eaa-8874-510e432ecc83"). InnerVolumeSpecName "kube-api-access-4mhp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:10.806668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.806585    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhp5\" (UniqueName: \"kubernetes.io/projected/52fb48d2-55e2-4eaa-8874-510e432ecc83-kube-api-access-4mhp5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:10.806668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:10.806617    5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fb48d2-55e2-4eaa-8874-510e432ecc83-metrics-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:11.304613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.304578    5196 generic.go:356] "Generic (PLEG): container finished" podID="52fb48d2-55e2-4eaa-8874-510e432ecc83" containerID="43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf" exitCode=0
Nov 03 20:41:11.305148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.304637    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf"
Nov 03 20:41:11.305148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.304662    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf" event={"ID":"52fb48d2-55e2-4eaa-8874-510e432ecc83","Type":"ContainerDied","Data":"43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf"}
Nov 03 20:41:11.305148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.304755    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-769788cd69-6rkhf" event={"ID":"52fb48d2-55e2-4eaa-8874-510e432ecc83","Type":"ContainerDied","Data":"175ef591caa51d4383eaa4d63c3d91617a0bc95da9ef2e92487739a1486a4af1"}
Nov 03 20:41:11.305148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.304792    5196 scope.go:122] "RemoveContainer" containerID="43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf"
Nov 03 20:41:11.314928 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.314910    5196 scope.go:122] "RemoveContainer" containerID="43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf"
Nov 03 20:41:11.315286 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:41:11.315265    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf\": container with ID starting with 43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf not found: ID does not exist" containerID="43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf"
Nov 03 20:41:11.315358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.315295    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf"} err="failed to get container status \"43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf\": rpc error: code = NotFound desc = could not find container \"43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf\": container with ID starting with 43d4afb1924bed8deaa053d3f2a8859968592c44109d548cf1da555139c70fdf not found: ID does not exist"
Nov 03 20:41:11.459992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.459959    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-769788cd69-6rkhf"]
Nov 03 20:41:11.506838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:11.506804    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-769788cd69-6rkhf"]
Nov 03 20:41:12.107851 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:12.107820    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fb48d2-55e2-4eaa-8874-510e432ecc83" path="/var/lib/kubelet/pods/52fb48d2-55e2-4eaa-8874-510e432ecc83/volumes"
Nov 03 20:41:18.485455 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:41:18.485409    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:41:28.030453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:28.030418    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:28.509365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:28.509327    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:39.788293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:39.788257    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:39.979607 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:39.979549    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:40.148408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:40.148370    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:41.623419 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:41.623382    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:41.779293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:41:41.779258    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:48.485592 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:41:48.485549    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:42:06.227278 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:06.227240    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.426998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:06.426960    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.786499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:06.786464    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.993840 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:06.993801    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:07.145623 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:07.145578    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:07.311780 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:07.311735    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:07.537061 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:07.536966    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:07.697884 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:07.697841    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:07.850243 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:07.850207    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:08.001261 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:08.001225    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:08.161462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:08.161356    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:08.320446 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:08.320413    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:08.519589 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:08.519496    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:09.017484 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:09.017446    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:10.885430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:10.885383    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:11.071507 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:11.071473    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:11.230821 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:11.230726    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:11.881594 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:11.881538    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:12.093591 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:12.093550    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:12.265829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:12.265714    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:13.793847 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:13.793807    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.019270 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.019235    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.176918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.176882    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.380943 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.380902    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.538451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.538362    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.718838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.718799    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.768999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.768960    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"]
Nov 03 20:42:14.769594 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.769578    5196 cpu_manager.go:414] "containerMap: removing container" podUID="52fb48d2-55e2-4eaa-8874-510e432ecc83" containerName="manager"
Nov 03 20:42:14.769654 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.769596    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fb48d2-55e2-4eaa-8874-510e432ecc83" containerName="manager"
Nov 03 20:42:14.769698 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.769689    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fb48d2-55e2-4eaa-8874-510e432ecc83" containerName="manager"
Nov 03 20:42:14.773348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.773332    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:14.826307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.826276    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"]
Nov 03 20:42:14.869347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.869305    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc7tm\" (UniqueName: \"kubernetes.io/projected/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-kube-api-access-gc7tm\") pod \"payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f\" (UID: \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:14.869347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.869362    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f\" (UID: \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:14.869347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.870015    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-1ec892\""
Nov 03 20:42:14.912947 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.912910    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"]
Nov 03 20:42:14.917276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.917245    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:14.929731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.929707    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-1ec892-dockercfg-xbkfb\""
Nov 03 20:42:14.971498 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.971080    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7tm\" (UniqueName: \"kubernetes.io/projected/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-kube-api-access-gc7tm\") pod \"payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f\" (UID: \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:14.971698 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.971542    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f\" (UID: \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:14.972228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.972196    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"]
Nov 03 20:42:14.972462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:14.972435    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f\" (UID: \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:15.037070 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.032184    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7tm\" (UniqueName: \"kubernetes.io/projected/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-kube-api-access-gc7tm\") pod \"payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f\" (UID: \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:15.072794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.072754    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjm5\" (UniqueName: \"kubernetes.io/projected/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-kube-api-access-vvjm5\") pod \"payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9\" (UID: \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:15.072977 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.072820    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9\" (UID: \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:15.087724 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.087634    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:15.173887 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.173841    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjm5\" (UniqueName: \"kubernetes.io/projected/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-kube-api-access-vvjm5\") pod \"payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9\" (UID: \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:15.174083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.173928    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9\" (UID: \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:15.174655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.174631    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9\" (UID: \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:15.228324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.228285    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjm5\" (UniqueName: \"kubernetes.io/projected/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-kube-api-access-vvjm5\") pod \"payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9\" (UID: \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:15.232183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.232157    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:15.464609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.464525    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"]
Nov 03 20:42:15.469871 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:15.469841    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33d4d1c_a8db_45f3_9796_b6f2c5ee4859.slice/crio-7d4cd296f9a43e7035d8195133e67ed0ea355263df883c19ed0beda795476e45 WatchSource:0}: Error finding container 7d4cd296f9a43e7035d8195133e67ed0ea355263df883c19ed0beda795476e45: Status 404 returned error can't find the container with id 7d4cd296f9a43e7035d8195133e67ed0ea355263df883c19ed0beda795476e45
Nov 03 20:42:15.607558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.607517    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"]
Nov 03 20:42:15.612779 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:15.612744    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f838b11_fd4b_492c_bf4e_15a5a0bf4208.slice/crio-65321dec21a55cd7edc93c1f9d9833ac873f3b25dcdacbb2fd51063952552b09 WatchSource:0}: Error finding container 65321dec21a55cd7edc93c1f9d9833ac873f3b25dcdacbb2fd51063952552b09: Status 404 returned error can't find the container with id 65321dec21a55cd7edc93c1f9d9833ac873f3b25dcdacbb2fd51063952552b09
Nov 03 20:42:15.709198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.709151    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f" event={"ID":"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859","Type":"ContainerStarted","Data":"ab8034821ef8c5679dd17813ff9b0f274b69a2421ab6e034d361f02ff2da4932"}
Nov 03 20:42:15.709395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.709211    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f" event={"ID":"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859","Type":"ContainerStarted","Data":"7d4cd296f9a43e7035d8195133e67ed0ea355263df883c19ed0beda795476e45"}
Nov 03 20:42:15.709395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.709231    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:15.710882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.710860    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9" event={"ID":"6f838b11-fd4b-492c-bf4e-15a5a0bf4208","Type":"ContainerStarted","Data":"d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8"}
Nov 03 20:42:15.711014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.710898    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9" event={"ID":"6f838b11-fd4b-492c-bf4e-15a5a0bf4208","Type":"ContainerStarted","Data":"65321dec21a55cd7edc93c1f9d9833ac873f3b25dcdacbb2fd51063952552b09"}
Nov 03 20:42:15.711014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.710978    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:15.819244 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.819192    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f" podStartSLOduration=1.819175017 podStartE2EDuration="1.819175017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:15.816872971 +0000 UTC m=+775.820831719" watchObservedRunningTime="2026-11-03 20:42:15.819175017 +0000 UTC m=+775.823133735"
Nov 03 20:42:15.926686 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:15.926637    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9" podStartSLOduration=1.9266243950000002 podStartE2EDuration="1.926624395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:15.924408653 +0000 UTC m=+775.928367398" watchObservedRunningTime="2026-11-03 20:42:15.926624395 +0000 UTC m=+775.930583119"
Nov 03 20:42:16.868708 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:16.868678    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:17.140500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:17.140418    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:17.305058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:17.305027    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:17.461636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:17.461542    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:17.627400 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:17.627363    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:18.485263 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:42:18.485226    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:42:18.945868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:18.945834    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:19.186039 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.185999    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:19.366472 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.366438    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:19.513143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.513084    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:19.704772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.704682    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"]
Nov 03 20:42:19.708836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.708810    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:19.720471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.720444    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt\" (UID: \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:19.720641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.720530    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wns8\" (UniqueName: \"kubernetes.io/projected/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-kube-api-access-8wns8\") pod \"payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt\" (UID: \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:19.805765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.805723    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b119f2\""
Nov 03 20:42:19.806005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.805750    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b119f2-dockercfg-gd8hw\""
Nov 03 20:42:19.808903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.808872    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"]
Nov 03 20:42:19.821424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.821380    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt\" (UID: \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:19.821638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.821499    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wns8\" (UniqueName: \"kubernetes.io/projected/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-kube-api-access-8wns8\") pod \"payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt\" (UID: \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:19.822190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.822171    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt\" (UID: \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:19.858904 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.858867    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"]
Nov 03 20:42:19.863266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.863246    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:19.879457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.879423    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wns8\" (UniqueName: \"kubernetes.io/projected/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-kube-api-access-8wns8\") pod \"payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt\" (UID: \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:19.916002 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.915970    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"]
Nov 03 20:42:19.923007 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.922971    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzxn\" (UniqueName: \"kubernetes.io/projected/b092dd7c-4650-406a-b0b5-216381fd4a07-kube-api-access-4rzxn\") pod \"payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw\" (UID: \"b092dd7c-4650-406a-b0b5-216381fd4a07\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:19.923265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:19.923180    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092dd7c-4650-406a-b0b5-216381fd4a07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw\" (UID: \"b092dd7c-4650-406a-b0b5-216381fd4a07\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:20.023738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.023632    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:20.023906 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.023850    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzxn\" (UniqueName: \"kubernetes.io/projected/b092dd7c-4650-406a-b0b5-216381fd4a07-kube-api-access-4rzxn\") pod \"payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw\" (UID: \"b092dd7c-4650-406a-b0b5-216381fd4a07\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:20.023973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.023950    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092dd7c-4650-406a-b0b5-216381fd4a07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw\" (UID: \"b092dd7c-4650-406a-b0b5-216381fd4a07\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:20.024660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.024634    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092dd7c-4650-406a-b0b5-216381fd4a07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw\" (UID: \"b092dd7c-4650-406a-b0b5-216381fd4a07\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:20.079483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.079455    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzxn\" (UniqueName: \"kubernetes.io/projected/b092dd7c-4650-406a-b0b5-216381fd4a07-kube-api-access-4rzxn\") pod \"payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw\" (UID: \"b092dd7c-4650-406a-b0b5-216381fd4a07\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:20.177956 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.177920    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:20.345643 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.345609    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:20.399143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.399105    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"]
Nov 03 20:42:20.404669 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:20.404638    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e873ca_6f50_4ad7_b052_74ffdbcd72cc.slice/crio-d16392ed3a06ed971f4c3b69d6411c5c60fb60570a631e21aa6f001759f9c20a WatchSource:0}: Error finding container d16392ed3a06ed971f4c3b69d6411c5c60fb60570a631e21aa6f001759f9c20a: Status 404 returned error can't find the container with id d16392ed3a06ed971f4c3b69d6411c5c60fb60570a631e21aa6f001759f9c20a
Nov 03 20:42:20.557951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.557915    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"]
Nov 03 20:42:20.563386 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:20.563354    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb092dd7c_4650_406a_b0b5_216381fd4a07.slice/crio-b2bc738000e96617281cfeb1b116c5fb1fdaa19ee796ab6278b1de8ee14dbd3c WatchSource:0}: Error finding container b2bc738000e96617281cfeb1b116c5fb1fdaa19ee796ab6278b1de8ee14dbd3c: Status 404 returned error can't find the container with id b2bc738000e96617281cfeb1b116c5fb1fdaa19ee796ab6278b1de8ee14dbd3c
Nov 03 20:42:20.751004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.750970    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt" event={"ID":"51e873ca-6f50-4ad7-b052-74ffdbcd72cc","Type":"ContainerStarted","Data":"0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89"}
Nov 03 20:42:20.751269 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.751030    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt" event={"ID":"51e873ca-6f50-4ad7-b052-74ffdbcd72cc","Type":"ContainerStarted","Data":"d16392ed3a06ed971f4c3b69d6411c5c60fb60570a631e21aa6f001759f9c20a"}
Nov 03 20:42:20.751269 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.751140    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:20.752598 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.752570    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw" event={"ID":"b092dd7c-4650-406a-b0b5-216381fd4a07","Type":"ContainerStarted","Data":"4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a"}
Nov 03 20:42:20.752711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.752619    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw" event={"ID":"b092dd7c-4650-406a-b0b5-216381fd4a07","Type":"ContainerStarted","Data":"b2bc738000e96617281cfeb1b116c5fb1fdaa19ee796ab6278b1de8ee14dbd3c"}
Nov 03 20:42:20.752711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.752685    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:20.858393 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.858276    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt" podStartSLOduration=1.8582614400000002 podStartE2EDuration="1.85826144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:20.857907669 +0000 UTC m=+780.861866416" watchObservedRunningTime="2026-11-03 20:42:20.85826144 +0000 UTC m=+780.862220164"
Nov 03 20:42:20.972629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:20.972565    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw" podStartSLOduration=1.972550273 podStartE2EDuration="1.972550273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:20.969479726 +0000 UTC m=+780.973438831" watchObservedRunningTime="2026-11-03 20:42:20.972550273 +0000 UTC m=+780.976508997"
Nov 03 20:42:21.776539 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:21.776506    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:21.973335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:21.973301    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.170899 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.170860    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"]
Nov 03 20:42:22.175146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.175083    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:22.228034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.227996    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"]
Nov 03 20:42:22.245000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.244957    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757404-5bf1-4e94-ba40-2c8ce0d83959-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5\" (UID: \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:22.245234 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.245037    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vh79\" (UniqueName: \"kubernetes.io/projected/3a757404-5bf1-4e94-ba40-2c8ce0d83959-kube-api-access-7vh79\") pod \"payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5\" (UID: \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:22.270959 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.270929    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a19980\""
Nov 03 20:42:22.271186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.270959    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a19980-dockercfg-5pb55\""
Nov 03 20:42:22.320609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.320562    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"]
Nov 03 20:42:22.324707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.324679    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:22.345902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.345868    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757404-5bf1-4e94-ba40-2c8ce0d83959-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5\" (UID: \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:22.346061 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.345916    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmps\" (UniqueName: \"kubernetes.io/projected/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-kube-api-access-fqmps\") pod \"payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv\" (UID: \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:22.346061 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.345965    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vh79\" (UniqueName: \"kubernetes.io/projected/3a757404-5bf1-4e94-ba40-2c8ce0d83959-kube-api-access-7vh79\") pod \"payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5\" (UID: \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:22.346193 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.346058    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv\" (UID: \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:22.346649 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.346631    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757404-5bf1-4e94-ba40-2c8ce0d83959-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5\" (UID: \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:22.372327 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.372291    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"]
Nov 03 20:42:22.402347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.402311    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vh79\" (UniqueName: \"kubernetes.io/projected/3a757404-5bf1-4e94-ba40-2c8ce0d83959-kube-api-access-7vh79\") pod \"payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5\" (UID: \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:22.447836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.447744    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmps\" (UniqueName: \"kubernetes.io/projected/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-kube-api-access-fqmps\") pod \"payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv\" (UID: \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:22.447836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.447813    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv\" (UID: \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:22.448571 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.448551    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv\" (UID: \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:22.490055 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.490019    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:22.501649 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.501625    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmps\" (UniqueName: \"kubernetes.io/projected/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-kube-api-access-fqmps\") pod \"payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv\" (UID: \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:22.639479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.639444    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:22.659754 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.659718    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.810320 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.810279    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.863548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:22.863510    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"]
Nov 03 20:42:22.868141 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:22.868107    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a757404_5bf1_4e94_ba40_2c8ce0d83959.slice/crio-8a687c67d1f37dc4bd651119d4e57bdfd22e81d89641a38770de80f38dc9a65b WatchSource:0}: Error finding container 8a687c67d1f37dc4bd651119d4e57bdfd22e81d89641a38770de80f38dc9a65b: Status 404 returned error can't find the container with id 8a687c67d1f37dc4bd651119d4e57bdfd22e81d89641a38770de80f38dc9a65b
Nov 03 20:42:23.004396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.004302    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:23.018826 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.018786    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"]
Nov 03 20:42:23.024369 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:23.024338    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbfd999_a314_42bd_9c1d_b8bcd1f7e699.slice/crio-e7f405e10e8bc64cffea6cd9e660c0d6ac2cbc0a8e9024b58bbd0e5c91d4d993 WatchSource:0}: Error finding container e7f405e10e8bc64cffea6cd9e660c0d6ac2cbc0a8e9024b58bbd0e5c91d4d993: Status 404 returned error can't find the container with id e7f405e10e8bc64cffea6cd9e660c0d6ac2cbc0a8e9024b58bbd0e5c91d4d993
Nov 03 20:42:23.771829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.771790    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5" event={"ID":"3a757404-5bf1-4e94-ba40-2c8ce0d83959","Type":"ContainerStarted","Data":"558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a"}
Nov 03 20:42:23.771829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.771835    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5" event={"ID":"3a757404-5bf1-4e94-ba40-2c8ce0d83959","Type":"ContainerStarted","Data":"8a687c67d1f37dc4bd651119d4e57bdfd22e81d89641a38770de80f38dc9a65b"}
Nov 03 20:42:23.772139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.772016    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:23.773370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.773343    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv" event={"ID":"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699","Type":"ContainerStarted","Data":"9a782d2974c7564446c495d7142eb96145768ff315655f4b04d0d8766cb32aeb"}
Nov 03 20:42:23.773508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.773386    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv" event={"ID":"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699","Type":"ContainerStarted","Data":"e7f405e10e8bc64cffea6cd9e660c0d6ac2cbc0a8e9024b58bbd0e5c91d4d993"}
Nov 03 20:42:23.773508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.773413    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:23.885914 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.885861    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5" podStartSLOduration=1.885845422 podStartE2EDuration="1.885845422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:23.883669797 +0000 UTC m=+783.887628541" watchObservedRunningTime="2026-11-03 20:42:23.885845422 +0000 UTC m=+783.889804145"
Nov 03 20:42:23.991062 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:23.991007    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv" podStartSLOduration=1.990992962 podStartE2EDuration="1.990992962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:23.990006494 +0000 UTC m=+783.993965255" watchObservedRunningTime="2026-11-03 20:42:23.990992962 +0000 UTC m=+783.994951685"
Nov 03 20:42:24.305751 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.305717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:24.513550 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.513516    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:24.654751 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.654708    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:42:24.693769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.693723    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"]
Nov 03 20:42:24.702406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.702375    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:24.755153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.753972    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-fd9afe-dockercfg-vpkp2\""
Nov 03 20:42:24.755153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.754183    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-fd9afe\""
Nov 03 20:42:24.760115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.757701    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"]
Nov 03 20:42:24.784117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.779721    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dng\" (UniqueName: \"kubernetes.io/projected/ad0688f5-93e5-4dde-b79c-8298e1843c23-kube-api-access-t6dng\") pod \"payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z\" (UID: \"ad0688f5-93e5-4dde-b79c-8298e1843c23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:24.784117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.779846    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0688f5-93e5-4dde-b79c-8298e1843c23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z\" (UID: \"ad0688f5-93e5-4dde-b79c-8298e1843c23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:24.875968 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.875926    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"]
Nov 03 20:42:24.880401 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.880368    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:24.880854 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.880821    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dng\" (UniqueName: \"kubernetes.io/projected/ad0688f5-93e5-4dde-b79c-8298e1843c23-kube-api-access-t6dng\") pod \"payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z\" (UID: \"ad0688f5-93e5-4dde-b79c-8298e1843c23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:24.881028 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.881006    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0688f5-93e5-4dde-b79c-8298e1843c23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z\" (UID: \"ad0688f5-93e5-4dde-b79c-8298e1843c23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:24.881882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.881858    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0688f5-93e5-4dde-b79c-8298e1843c23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z\" (UID: \"ad0688f5-93e5-4dde-b79c-8298e1843c23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:24.939786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.939710    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"]
Nov 03 20:42:24.945163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.945136    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dng\" (UniqueName: \"kubernetes.io/projected/ad0688f5-93e5-4dde-b79c-8298e1843c23-kube-api-access-t6dng\") pod \"payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z\" (UID: \"ad0688f5-93e5-4dde-b79c-8298e1843c23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:24.982433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.982386    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34243db-4751-4a72-ad70-4595e94a4af0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n\" (UID: \"e34243db-4751-4a72-ad70-4595e94a4af0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:24.982433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:24.982443    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4skh\" (UniqueName: \"kubernetes.io/projected/e34243db-4751-4a72-ad70-4595e94a4af0-kube-api-access-d4skh\") pod \"payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n\" (UID: \"e34243db-4751-4a72-ad70-4595e94a4af0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:25.044119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.044066    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:25.083976 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.083935    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4skh\" (UniqueName: \"kubernetes.io/projected/e34243db-4751-4a72-ad70-4595e94a4af0-kube-api-access-d4skh\") pod \"payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n\" (UID: \"e34243db-4751-4a72-ad70-4595e94a4af0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:25.084168 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.084123    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34243db-4751-4a72-ad70-4595e94a4af0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n\" (UID: \"e34243db-4751-4a72-ad70-4595e94a4af0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:25.084885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.084861    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34243db-4751-4a72-ad70-4595e94a4af0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n\" (UID: \"e34243db-4751-4a72-ad70-4595e94a4af0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:25.138660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.138623    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4skh\" (UniqueName: \"kubernetes.io/projected/e34243db-4751-4a72-ad70-4595e94a4af0-kube-api-access-d4skh\") pod \"payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n\" (UID: \"e34243db-4751-4a72-ad70-4595e94a4af0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:25.197636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.197553    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:25.421208 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.421171    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"]
Nov 03 20:42:25.427594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:25.427563    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0688f5_93e5_4dde_b79c_8298e1843c23.slice/crio-e4ce615001110edd9c5543d6e5749de1c1ccd9dbd6857688fdb8fa971062942a WatchSource:0}: Error finding container e4ce615001110edd9c5543d6e5749de1c1ccd9dbd6857688fdb8fa971062942a: Status 404 returned error can't find the container with id e4ce615001110edd9c5543d6e5749de1c1ccd9dbd6857688fdb8fa971062942a
Nov 03 20:42:25.575614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.575579    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"]
Nov 03 20:42:25.581022 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:25.580992    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34243db_4751_4a72_ad70_4595e94a4af0.slice/crio-b4a253e57db05c55cac36f05825a9094ff3d8f132497c3f63241137fc709b514 WatchSource:0}: Error finding container b4a253e57db05c55cac36f05825a9094ff3d8f132497c3f63241137fc709b514: Status 404 returned error can't find the container with id b4a253e57db05c55cac36f05825a9094ff3d8f132497c3f63241137fc709b514
Nov 03 20:42:25.624566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.624541    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:25.842623 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.842588    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:25.859628 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.859594    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z" event={"ID":"ad0688f5-93e5-4dde-b79c-8298e1843c23","Type":"ContainerStarted","Data":"277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22"}
Nov 03 20:42:25.859832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.859680    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z" event={"ID":"ad0688f5-93e5-4dde-b79c-8298e1843c23","Type":"ContainerStarted","Data":"e4ce615001110edd9c5543d6e5749de1c1ccd9dbd6857688fdb8fa971062942a"}
Nov 03 20:42:25.859832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.859710    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:25.861369 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.861343    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n" event={"ID":"e34243db-4751-4a72-ad70-4595e94a4af0","Type":"ContainerStarted","Data":"b9499b5f7f9510c529c2fb6babcd285ce91adb8b5338e24cdc533fa52de5a5dd"}
Nov 03 20:42:25.861493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.861391    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n" event={"ID":"e34243db-4751-4a72-ad70-4595e94a4af0","Type":"ContainerStarted","Data":"b4a253e57db05c55cac36f05825a9094ff3d8f132497c3f63241137fc709b514"}
Nov 03 20:42:25.861493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.861439    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:25.970194 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.970137    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z" podStartSLOduration=1.970121417 podStartE2EDuration="1.970121417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:25.96829494 +0000 UTC m=+785.972253689" watchObservedRunningTime="2026-11-03 20:42:25.970121417 +0000 UTC m=+785.974080141"
Nov 03 20:42:25.998891 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:25.998858    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:26.078008 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:26.077953    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n" podStartSLOduration=2.077938254 podStartE2EDuration="2.077938254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:26.076261834 +0000 UTC m=+786.080220576" watchObservedRunningTime="2026-11-03 20:42:26.077938254 +0000 UTC m=+786.081896978"
Nov 03 20:42:26.719557 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:26.719530    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 20:42:26.719744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:26.719599    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 20:42:26.830364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:26.830330    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:26.997033 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:26.996936    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:27.174845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.174808    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:27.221557 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.221526    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"]
Nov 03 20:42:27.227893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.227857    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:27.277887 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.277786    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-41aa21-dockercfg-n6vj9\""
Nov 03 20:42:27.278559 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.278533    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-41aa21\""
Nov 03 20:42:27.278754 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.278715    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"]
Nov 03 20:42:27.321153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.308986    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mwrj\" (UniqueName: \"kubernetes.io/projected/b6df122a-3baa-4fba-b3b0-4e6521e682d8-kube-api-access-6mwrj\") pod \"payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8\" (UID: \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:27.321153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.309117    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6df122a-3baa-4fba-b3b0-4e6521e682d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8\" (UID: \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:27.327796 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.327760    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:27.368312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.368277    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"]
Nov 03 20:42:27.372279 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.372261    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:27.410536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.410503    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43f5087-4ed9-4e95-bf72-57432ed7681c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g\" (UID: \"e43f5087-4ed9-4e95-bf72-57432ed7681c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:27.410713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.410552    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5crq\" (UniqueName: \"kubernetes.io/projected/e43f5087-4ed9-4e95-bf72-57432ed7681c-kube-api-access-v5crq\") pod \"payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g\" (UID: \"e43f5087-4ed9-4e95-bf72-57432ed7681c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:27.410713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.410625    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mwrj\" (UniqueName: \"kubernetes.io/projected/b6df122a-3baa-4fba-b3b0-4e6521e682d8-kube-api-access-6mwrj\") pod \"payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8\" (UID: \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:27.410713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.410681    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6df122a-3baa-4fba-b3b0-4e6521e682d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8\" (UID: \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:27.411394 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.411373    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6df122a-3baa-4fba-b3b0-4e6521e682d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8\" (UID: \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:27.425743 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.425713    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"]
Nov 03 20:42:27.466165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.466124    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mwrj\" (UniqueName: \"kubernetes.io/projected/b6df122a-3baa-4fba-b3b0-4e6521e682d8-kube-api-access-6mwrj\") pod \"payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8\" (UID: \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:27.511670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.511622    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43f5087-4ed9-4e95-bf72-57432ed7681c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g\" (UID: \"e43f5087-4ed9-4e95-bf72-57432ed7681c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:27.511857 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.511684    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5crq\" (UniqueName: \"kubernetes.io/projected/e43f5087-4ed9-4e95-bf72-57432ed7681c-kube-api-access-v5crq\") pod \"payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g\" (UID: \"e43f5087-4ed9-4e95-bf72-57432ed7681c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:27.512445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.512425    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43f5087-4ed9-4e95-bf72-57432ed7681c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g\" (UID: \"e43f5087-4ed9-4e95-bf72-57432ed7681c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:27.543382 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.543290    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:27.573933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.573902    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5crq\" (UniqueName: \"kubernetes.io/projected/e43f5087-4ed9-4e95-bf72-57432ed7681c-kube-api-access-v5crq\") pod \"payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g\" (UID: \"e43f5087-4ed9-4e95-bf72-57432ed7681c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:27.686490 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.686454    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:27.919610 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.919572    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"]
Nov 03 20:42:27.925205 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:27.925177    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6df122a_3baa_4fba_b3b0_4e6521e682d8.slice/crio-da5994d5b3621fb608e2a7331f8b5245231ea589d986bda1b3720932d06532ff WatchSource:0}: Error finding container da5994d5b3621fb608e2a7331f8b5245231ea589d986bda1b3720932d06532ff: Status 404 returned error can't find the container with id da5994d5b3621fb608e2a7331f8b5245231ea589d986bda1b3720932d06532ff
Nov 03 20:42:27.958741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:27.958694    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:28.060882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.060850    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"]
Nov 03 20:42:28.066023 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:28.065996    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43f5087_4ed9_4e95_bf72_57432ed7681c.slice/crio-e90b3d97ff0c5d9a648a7e68c49988860913b2e42b0e491ff7b3e3a172194b95 WatchSource:0}: Error finding container e90b3d97ff0c5d9a648a7e68c49988860913b2e42b0e491ff7b3e3a172194b95: Status 404 returned error can't find the container with id e90b3d97ff0c5d9a648a7e68c49988860913b2e42b0e491ff7b3e3a172194b95
Nov 03 20:42:28.159857 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.159807    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:28.316443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.316329    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:28.882957 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.882924    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g" event={"ID":"e43f5087-4ed9-4e95-bf72-57432ed7681c","Type":"ContainerStarted","Data":"7ebb7666397c9416561479b2e12563ad152873ac8025dbee2777a3b0f9b5b383"}
Nov 03 20:42:28.883160 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.882987    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:28.883160 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.883011    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g" event={"ID":"e43f5087-4ed9-4e95-bf72-57432ed7681c","Type":"ContainerStarted","Data":"e90b3d97ff0c5d9a648a7e68c49988860913b2e42b0e491ff7b3e3a172194b95"}
Nov 03 20:42:28.884687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.884659    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8" event={"ID":"b6df122a-3baa-4fba-b3b0-4e6521e682d8","Type":"ContainerStarted","Data":"359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a"}
Nov 03 20:42:28.884813 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.884699    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8" event={"ID":"b6df122a-3baa-4fba-b3b0-4e6521e682d8","Type":"ContainerStarted","Data":"da5994d5b3621fb608e2a7331f8b5245231ea589d986bda1b3720932d06532ff"}
Nov 03 20:42:28.884813 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.884721    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:28.994065 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:28.994012    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g" podStartSLOduration=1.9939983049999999 podStartE2EDuration="1.993998305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:28.992817549 +0000 UTC m=+788.996776275" watchObservedRunningTime="2026-11-03 20:42:28.993998305 +0000 UTC m=+788.997957029"
Nov 03 20:42:29.101321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:29.101252    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8" podStartSLOduration=2.101233103 podStartE2EDuration="2.101233103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:29.099910165 +0000 UTC m=+789.103868914" watchObservedRunningTime="2026-11-03 20:42:29.101233103 +0000 UTC m=+789.105191827"
Nov 03 20:42:31.761976 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:31.761940    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 20:42:31.762405 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:31.762020    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 20:42:33.150716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:33.150674    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:42:33.472083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:33.471996    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:34.852044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:34.852007    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 20:42:34.852556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:34.852109    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 20:42:36.872301 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:36.872274    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 20:42:36.872742 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:36.872341    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 20:42:38.593260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:38.593219    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:38.753956 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:38.753914    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:38.909594 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:38.909496    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:39.895177 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:39.895138    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 20:42:39.895599 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:39.895227    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 20:42:40.117069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:40.117037    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:41.009033 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.008993    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:41.188655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.188596    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:41.405489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.405452    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"]
Nov 03 20:42:41.409883 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.409861    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:41.462114 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.462070    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"]
Nov 03 20:42:41.506610 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.506574    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-984cbc-dockercfg-pd5xk\""
Nov 03 20:42:41.509916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.509883    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-984cbc\""
Nov 03 20:42:41.565272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.565226    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"]
Nov 03 20:42:41.569928 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.569901    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:41.570713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.570670    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwztq\" (UniqueName: \"kubernetes.io/projected/46df0c79-9c14-40f7-9ece-6577e91d8dfc-kube-api-access-lwztq\") pod \"payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt\" (UID: \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:41.570882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.570856    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46df0c79-9c14-40f7-9ece-6577e91d8dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt\" (UID: \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:41.622814 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.622781    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"]
Nov 03 20:42:41.672679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.672582    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46df0c79-9c14-40f7-9ece-6577e91d8dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt\" (UID: \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:41.672850 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.672701    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd9390b-e208-479c-b6c7-a52c4ddbff51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l\" (UID: \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:41.672850 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.672739    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lxc\" (UniqueName: \"kubernetes.io/projected/3dd9390b-e208-479c-b6c7-a52c4ddbff51-kube-api-access-w7lxc\") pod \"payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l\" (UID: \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:41.672937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.672857    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwztq\" (UniqueName: \"kubernetes.io/projected/46df0c79-9c14-40f7-9ece-6577e91d8dfc-kube-api-access-lwztq\") pod \"payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt\" (UID: \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:41.673460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.673440    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46df0c79-9c14-40f7-9ece-6577e91d8dfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt\" (UID: \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:41.727169 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.727127    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwztq\" (UniqueName: \"kubernetes.io/projected/46df0c79-9c14-40f7-9ece-6577e91d8dfc-kube-api-access-lwztq\") pod \"payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt\" (UID: \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:41.774459 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.774417    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd9390b-e208-479c-b6c7-a52c4ddbff51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l\" (UID: \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:41.774459 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.774459    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lxc\" (UniqueName: \"kubernetes.io/projected/3dd9390b-e208-479c-b6c7-a52c4ddbff51-kube-api-access-w7lxc\") pod \"payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l\" (UID: \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:41.775286 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.775259    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd9390b-e208-479c-b6c7-a52c4ddbff51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l\" (UID: \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:41.829328 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.829298    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lxc\" (UniqueName: \"kubernetes.io/projected/3dd9390b-e208-479c-b6c7-a52c4ddbff51-kube-api-access-w7lxc\") pod \"payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l\" (UID: \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:41.887784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:41.887747    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:42.025293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.025260    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:42.267130 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.267033    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"]
Nov 03 20:42:42.271857 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:42.271825    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd9390b_e208_479c_b6c7_a52c4ddbff51.slice/crio-95cf9882123ef1ba2f035c3e503875724e9c875921c67f567c78cdf9051ee902 WatchSource:0}: Error finding container 95cf9882123ef1ba2f035c3e503875724e9c875921c67f567c78cdf9051ee902: Status 404 returned error can't find the container with id 95cf9882123ef1ba2f035c3e503875724e9c875921c67f567c78cdf9051ee902
Nov 03 20:42:42.400525 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.400491    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"]
Nov 03 20:42:42.405705 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:42.405679    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46df0c79_9c14_40f7_9ece_6577e91d8dfc.slice/crio-9045b29075c8df57c7beddad4b98700840df0e7d41aa4d341fc8e094961eca0e WatchSource:0}: Error finding container 9045b29075c8df57c7beddad4b98700840df0e7d41aa4d341fc8e094961eca0e: Status 404 returned error can't find the container with id 9045b29075c8df57c7beddad4b98700840df0e7d41aa4d341fc8e094961eca0e
Nov 03 20:42:42.984737 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.984703    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt" event={"ID":"46df0c79-9c14-40f7-9ece-6577e91d8dfc","Type":"ContainerStarted","Data":"99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4"}
Nov 03 20:42:42.984970 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.984765    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:42.984970 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.984793    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt" event={"ID":"46df0c79-9c14-40f7-9ece-6577e91d8dfc","Type":"ContainerStarted","Data":"9045b29075c8df57c7beddad4b98700840df0e7d41aa4d341fc8e094961eca0e"}
Nov 03 20:42:42.986480 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.986451    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l" event={"ID":"3dd9390b-e208-479c-b6c7-a52c4ddbff51","Type":"ContainerStarted","Data":"8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595"}
Nov 03 20:42:42.986623 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.986501    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l" event={"ID":"3dd9390b-e208-479c-b6c7-a52c4ddbff51","Type":"ContainerStarted","Data":"95cf9882123ef1ba2f035c3e503875724e9c875921c67f567c78cdf9051ee902"}
Nov 03 20:42:42.986623 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:42.986527    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:43.095396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.095345    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt" podStartSLOduration=2.09533015 podStartE2EDuration="2.09533015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:43.093205939 +0000 UTC m=+803.097164662" watchObservedRunningTime="2026-11-03 20:42:43.09533015 +0000 UTC m=+803.099288878"
Nov 03 20:42:43.200580 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.200532    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l" podStartSLOduration=2.200519539 podStartE2EDuration="2.200519539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:43.199355375 +0000 UTC m=+803.203314120" watchObservedRunningTime="2026-11-03 20:42:43.200519539 +0000 UTC m=+803.204478262"
Nov 03 20:42:43.478951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.478898    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:43.653139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.653063    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:43.867269 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.867237    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"]
Nov 03 20:42:43.871341 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.871324    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:43.922047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.921365    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-fe2825\""
Nov 03 20:42:43.922261 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.922130    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-fe2825-dockercfg-r9lsr\""
Nov 03 20:42:43.926672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.926640    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"]
Nov 03 20:42:43.998535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.998501    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxk7s\" (UniqueName: \"kubernetes.io/projected/6a6e35fd-2d40-46b4-8655-b20c54443217-kube-api-access-sxk7s\") pod \"payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8\" (UID: \"6a6e35fd-2d40-46b4-8655-b20c54443217\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:43.998535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:43.998536    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6e35fd-2d40-46b4-8655-b20c54443217-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8\" (UID: \"6a6e35fd-2d40-46b4-8655-b20c54443217\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:44.020946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.020912    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"]
Nov 03 20:42:44.025064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.025037    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:44.079598 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.079569    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"]
Nov 03 20:42:44.095878 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.095845    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:44.100020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.099995    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxk7s\" (UniqueName: \"kubernetes.io/projected/6a6e35fd-2d40-46b4-8655-b20c54443217-kube-api-access-sxk7s\") pod \"payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8\" (UID: \"6a6e35fd-2d40-46b4-8655-b20c54443217\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:44.100179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.100024    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6e35fd-2d40-46b4-8655-b20c54443217-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8\" (UID: \"6a6e35fd-2d40-46b4-8655-b20c54443217\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:44.100179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.100064    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkgcs\" (UniqueName: \"kubernetes.io/projected/ecef5666-3f31-4093-b950-4a21b3f6a147-kube-api-access-mkgcs\") pod \"payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8\" (UID: \"ecef5666-3f31-4093-b950-4a21b3f6a147\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:44.100308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.100278    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecef5666-3f31-4093-b950-4a21b3f6a147-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8\" (UID: \"ecef5666-3f31-4093-b950-4a21b3f6a147\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:44.100819 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.100797    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6e35fd-2d40-46b4-8655-b20c54443217-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8\" (UID: \"6a6e35fd-2d40-46b4-8655-b20c54443217\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:44.154206 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.154129    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxk7s\" (UniqueName: \"kubernetes.io/projected/6a6e35fd-2d40-46b4-8655-b20c54443217-kube-api-access-sxk7s\") pod \"payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8\" (UID: \"6a6e35fd-2d40-46b4-8655-b20c54443217\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:44.185953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.185916    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:44.201595 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.201566    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecef5666-3f31-4093-b950-4a21b3f6a147-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8\" (UID: \"ecef5666-3f31-4093-b950-4a21b3f6a147\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:44.201702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.201685    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkgcs\" (UniqueName: \"kubernetes.io/projected/ecef5666-3f31-4093-b950-4a21b3f6a147-kube-api-access-mkgcs\") pod \"payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8\" (UID: \"ecef5666-3f31-4093-b950-4a21b3f6a147\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:44.202315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.202292    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecef5666-3f31-4093-b950-4a21b3f6a147-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8\" (UID: \"ecef5666-3f31-4093-b950-4a21b3f6a147\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:44.256214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.256185    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkgcs\" (UniqueName: \"kubernetes.io/projected/ecef5666-3f31-4093-b950-4a21b3f6a147-kube-api-access-mkgcs\") pod \"payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8\" (UID: \"ecef5666-3f31-4093-b950-4a21b3f6a147\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:44.338960 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.338919    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:44.562273 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.562186    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"]
Nov 03 20:42:44.566778 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:44.566751    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6e35fd_2d40_46b4_8655_b20c54443217.slice/crio-4387db8707206bc27f176aef69d388c13f12490d47d76b3236a586688e3b3627 WatchSource:0}: Error finding container 4387db8707206bc27f176aef69d388c13f12490d47d76b3236a586688e3b3627: Status 404 returned error can't find the container with id 4387db8707206bc27f176aef69d388c13f12490d47d76b3236a586688e3b3627
Nov 03 20:42:44.712627 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:44.712590    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"]
Nov 03 20:42:44.717927 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:44.717893    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecef5666_3f31_4093_b950_4a21b3f6a147.slice/crio-5157e1caa4596273c47bce87884185b772e4d115dcdb460a030147e80b2fa48f WatchSource:0}: Error finding container 5157e1caa4596273c47bce87884185b772e4d115dcdb460a030147e80b2fa48f: Status 404 returned error can't find the container with id 5157e1caa4596273c47bce87884185b772e4d115dcdb460a030147e80b2fa48f
Nov 03 20:42:45.002107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:45.002051    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8" event={"ID":"ecef5666-3f31-4093-b950-4a21b3f6a147","Type":"ContainerStarted","Data":"c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83"}
Nov 03 20:42:45.002315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:45.002130    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8" event={"ID":"ecef5666-3f31-4093-b950-4a21b3f6a147","Type":"ContainerStarted","Data":"5157e1caa4596273c47bce87884185b772e4d115dcdb460a030147e80b2fa48f"}
Nov 03 20:42:45.002315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:45.002161    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:45.003847 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:45.003819    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8" event={"ID":"6a6e35fd-2d40-46b4-8655-b20c54443217","Type":"ContainerStarted","Data":"d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08"}
Nov 03 20:42:45.003949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:45.003860    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:45.003949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:45.003880    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8" event={"ID":"6a6e35fd-2d40-46b4-8655-b20c54443217","Type":"ContainerStarted","Data":"4387db8707206bc27f176aef69d388c13f12490d47d76b3236a586688e3b3627"}
Nov 03 20:42:45.113613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:45.113555    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8" podStartSLOduration=2.113538812 podStartE2EDuration="2.113538812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:45.111215484 +0000 UTC m=+805.115174208" watchObservedRunningTime="2026-11-03 20:42:45.113538812 +0000 UTC m=+805.117497538"
Nov 03 20:42:45.221248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:45.221199    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8" podStartSLOduration=2.221185514 podStartE2EDuration="2.221185514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:45.219295053 +0000 UTC m=+805.223253796" watchObservedRunningTime="2026-11-03 20:42:45.221185514 +0000 UTC m=+805.225144241"
Nov 03 20:42:48.481818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:48.481778    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:42:48.490558 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:42:48.490528    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:42:49.229621 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:49.229588    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:49.394960 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:49.394923    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:49.556566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:49.556480    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:51.887279 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:51.887243    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:52.446072 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:52.446034    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:52.647433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:52.647395    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:53.996982 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:53.996892    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 20:42:53.997487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:53.997301    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 20:42:54.466417 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:54.466380    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:54.623267 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:54.623228    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:54.769591 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:54.769495    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:55.426684 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.426647    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"]
Nov 03 20:42:55.432492 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.432464    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.484351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.484304    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"]
Nov 03 20:42:55.520277 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.520243    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5af553-68f7-4fd1-830b-c334ec31edc0-tls-certs\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.520277 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.520279    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-tmp-dir\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.520505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.520387    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-model-cache\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.520505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.520409    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4bz\" (UniqueName: \"kubernetes.io/projected/5e5af553-68f7-4fd1-830b-c334ec31edc0-kube-api-access-6r4bz\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.520505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.520441    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-dshm\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.520670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.520583    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-home\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.527282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.527250    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"openshift-service-ca.crt\""
Nov 03 20:42:55.527531 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.527513    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"kube-root-ca.crt\""
Nov 03 20:42:55.528836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.528821    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"default-dockercfg-596g2\""
Nov 03 20:42:55.528893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.528851    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fd9afe\"/\"simulator-w0-fd9afe-kserve-self-signed-certs\""
Nov 03 20:42:55.543371 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.543338    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:55.621354 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621316    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-model-cache\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.621354 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621363    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4bz\" (UniqueName: \"kubernetes.io/projected/5e5af553-68f7-4fd1-830b-c334ec31edc0-kube-api-access-6r4bz\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.621629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621395    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-dshm\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.621629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621439    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-home\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.621629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621492    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5af553-68f7-4fd1-830b-c334ec31edc0-tls-certs\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.621629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621516    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-tmp-dir\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.621845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621820    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-model-cache\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.621978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621944    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-home\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.622117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.621994    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-tmp-dir\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.624044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.624016    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-dshm\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.624221 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.624205    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5af553-68f7-4fd1-830b-c334ec31edc0-tls-certs\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.676766 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.676688    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4bz\" (UniqueName: \"kubernetes.io/projected/5e5af553-68f7-4fd1-830b-c334ec31edc0-kube-api-access-6r4bz\") pod \"simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") " pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:55.745318 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:55.745286    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:56.014341 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:56.014269    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 20:42:56.014341 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:56.014337    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 20:42:56.119155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:56.119116    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"]
Nov 03 20:42:56.125802 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:42:56.125774    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e5af553_68f7_4fd1_830b_c334ec31edc0.slice/crio-77b64375713e21c7cb6eae2a43113cee64b5c291617d34ff576d8a8319d15efa WatchSource:0}: Error finding container 77b64375713e21c7cb6eae2a43113cee64b5c291617d34ff576d8a8319d15efa: Status 404 returned error can't find the container with id 77b64375713e21c7cb6eae2a43113cee64b5c291617d34ff576d8a8319d15efa
Nov 03 20:42:57.088967 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:57.088925    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" event={"ID":"5e5af553-68f7-4fd1-830b-c334ec31edc0","Type":"ContainerStarted","Data":"de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688"}
Nov 03 20:42:57.089451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:57.088979    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" event={"ID":"5e5af553-68f7-4fd1-830b-c334ec31edc0","Type":"ContainerStarted","Data":"77b64375713e21c7cb6eae2a43113cee64b5c291617d34ff576d8a8319d15efa"}
Nov 03 20:42:57.203052 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:57.202998    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podStartSLOduration=2.202985959 podStartE2EDuration="2.202985959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:57.202210735 +0000 UTC m=+817.206169477" watchObservedRunningTime="2026-11-03 20:42:57.202985959 +0000 UTC m=+817.206944739"
Nov 03 20:42:57.745929 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:57.745891    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:42:57.890117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:57.889738    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:42:59.429624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:59.429587    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:59.621465 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:59.621427    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:59.788986 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:42:59.788882    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:04.359975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:04.359936    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:05.745984 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:05.745953    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:43:05.759937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:05.759904    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:43:06.164526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:06.164492    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:43:09.080589 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:09.080544    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:09.427022 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:09.426925    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:43:13.646954 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:13.646918    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:14.225423 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:14.225369    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:14.384087 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:14.384040    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:16.679135 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:16.679076    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:17.304791 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:17.304760    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:17.470465 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:17.470424    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:18.485580 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:43:18.485532    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:43:18.689278 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.689238    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"]
Nov 03 20:43:18.694935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.694910    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.744307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.744200    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"]
Nov 03 20:43:18.779635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.779596    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-model-cache\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.779813 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.779669    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6z2\" (UniqueName: \"kubernetes.io/projected/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-kube-api-access-mr6z2\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.779813 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.779740    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-dshm\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.779813 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.779788    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-home\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.779927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.779840    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tls-certs\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.779927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.779902    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tmp-dir\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.787926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.787899    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"openshift-service-ca.crt\""
Nov 03 20:43:18.787926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.787912    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"kube-root-ca.crt\""
Nov 03 20:43:18.791009 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.790993    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"simulator-w1-41aa21-kserve-self-signed-certs\""
Nov 03 20:43:18.791009 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.790998    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"default-dockercfg-2nc52\""
Nov 03 20:43:18.797459 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.797436    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:18.881416 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.881378    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tls-certs\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.881609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.881444    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tmp-dir\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.881609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.881523    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-model-cache\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.881609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.881595    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6z2\" (UniqueName: \"kubernetes.io/projected/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-kube-api-access-mr6z2\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.881722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.881623    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-dshm\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.881722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.881670    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-home\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.882008 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.881972    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tmp-dir\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.882152 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.881983    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-model-cache\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.882152 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.882041    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-home\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.883891 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.883873    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-dshm\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.884303 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.884287    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tls-certs\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:18.936074 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:18.936028    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6z2\" (UniqueName: \"kubernetes.io/projected/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-kube-api-access-mr6z2\") pod \"simulator-w1-41aa21-kserve-5f6c4888d-l5kvs\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") " pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:19.011868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:19.011764    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:19.387031 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:19.386994    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"]
Nov 03 20:43:19.392449 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:19.392417    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:19.392924 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:43:19.392898    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f922216_cc27_48ab_99c2_e2dd7fc9ba3b.slice/crio-050ce90bed1406bfbdb8864a10fe696fb4ad275216c785d5052cbd99b0daf107 WatchSource:0}: Error finding container 050ce90bed1406bfbdb8864a10fe696fb4ad275216c785d5052cbd99b0daf107: Status 404 returned error can't find the container with id 050ce90bed1406bfbdb8864a10fe696fb4ad275216c785d5052cbd99b0daf107
Nov 03 20:43:20.249695 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:20.249655    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" event={"ID":"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b","Type":"ContainerStarted","Data":"3928a45ee9a2b53758cf56cc9e9b9a7ee53f5b3155986f5931c90f74957188f9"}
Nov 03 20:43:20.249695 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:20.249708    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" event={"ID":"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b","Type":"ContainerStarted","Data":"050ce90bed1406bfbdb8864a10fe696fb4ad275216c785d5052cbd99b0daf107"}
Nov 03 20:43:20.359313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:20.359258    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podStartSLOduration=2.3592422539999998 podStartE2EDuration="2.359242254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:20.356494699 +0000 UTC m=+840.360453448" watchObservedRunningTime="2026-11-03 20:43:20.359242254 +0000 UTC m=+840.363200978"
Nov 03 20:43:21.012741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:21.012703    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:24.176184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:24.176141    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:25.688560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:25.688520    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:43:29.012321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:29.012283    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:29.025059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:29.025033    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:29.272689 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:29.272591    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:29.330690 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:29.330661    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:43:33.043669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:33.043635    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:33.740904 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:33.740864    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:33.938749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:33.938717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:34.215948 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:34.215908    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:37.805190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:37.805159    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:39.152383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:39.152342    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:39.305691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:39.305597    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:39.454238 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:39.454133    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:39.703875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:39.703833    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:39.856719 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:39.856685    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:40.453435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:40.453387    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:42.458524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.458478    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"]
Nov 03 20:43:42.466005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.465971    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.515825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.515787    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"]
Nov 03 20:43:42.534634 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.534601    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-dshm\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.534634 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.534648    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-home\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.534873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.534667    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkhp\" (UniqueName: \"kubernetes.io/projected/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-kube-api-access-nrkhp\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.534917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.534899    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tls-certs\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.535062 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.535041    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tmp-dir\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.535159 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.535116    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-model-cache\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.537176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.537149    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:42.558272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.558235    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"openshift-service-ca.crt\""
Nov 03 20:43:42.558437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.558278    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"default-dockercfg-sx87j\""
Nov 03 20:43:42.558437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.558288    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"kube-root-ca.crt\""
Nov 03 20:43:42.562973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.562957    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"simulator-w5-1ec892-kserve-self-signed-certs\""
Nov 03 20:43:42.635874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.635839    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tls-certs\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.636058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.635941    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tmp-dir\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.636058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.635975    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-model-cache\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.636058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.636033    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-dshm\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.636235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.636070    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-home\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.636235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.636115    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkhp\" (UniqueName: \"kubernetes.io/projected/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-kube-api-access-nrkhp\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.636672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.636589    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tmp-dir\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.636672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.636608    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-home\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.636867 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.636845    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-model-cache\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.638859 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.638835    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tls-certs\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.639608 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.639587    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-dshm\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.690436 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.690394    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkhp\" (UniqueName: \"kubernetes.io/projected/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-kube-api-access-nrkhp\") pod \"simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") " pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:42.780343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:42.780256    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:43.156590 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:43.156555    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"]
Nov 03 20:43:43.162542 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:43:43.162512    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4d80abf_0dc2_4ba7_9bc6_3692275e9287.slice/crio-836ecf13ae83db7daa2fa006d19d7987c99c9806a2ff1d70d18ef4e26a30d8fb WatchSource:0}: Error finding container 836ecf13ae83db7daa2fa006d19d7987c99c9806a2ff1d70d18ef4e26a30d8fb: Status 404 returned error can't find the container with id 836ecf13ae83db7daa2fa006d19d7987c99c9806a2ff1d70d18ef4e26a30d8fb
Nov 03 20:43:43.169017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:43.168980    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:43.416647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:43.416549    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" event={"ID":"f4d80abf-0dc2-4ba7-9bc6-3692275e9287","Type":"ContainerStarted","Data":"1f9859b0a2e54dadb8dd05e73bea4bc6969084a35c79de1c7e6cb60009c41ce2"}
Nov 03 20:43:43.416647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:43.416593    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" event={"ID":"f4d80abf-0dc2-4ba7-9bc6-3692275e9287","Type":"ContainerStarted","Data":"836ecf13ae83db7daa2fa006d19d7987c99c9806a2ff1d70d18ef4e26a30d8fb"}
Nov 03 20:43:43.534037 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:43.533968    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podStartSLOduration=1.5339442490000001 podStartE2EDuration="1.533944249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:43.529826216 +0000 UTC m=+863.533784952" watchObservedRunningTime="2026-11-03 20:43:43.533944249 +0000 UTC m=+863.537902974"
Nov 03 20:43:43.780417 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:43.780332    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:44.280846 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:44.280811    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:45.764411 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:45.764373    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:48.485669 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:43:48.485633    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:43:49.270142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:49.270103    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:49.462922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:49.462887    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:49.835295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:49.835258    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:49.989240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:49.989180    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:50.415353 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:50.415315    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:43:52.100377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.100341    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"]
Nov 03 20:43:52.104652 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.104627    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.158035 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.157997    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"]
Nov 03 20:43:52.201226 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.201188    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"kube-root-ca.crt\""
Nov 03 20:43:52.201433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.201253    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"openshift-service-ca.crt\""
Nov 03 20:43:52.201433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.201343    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"simulator-w2-b119f2-kserve-self-signed-certs\""
Nov 03 20:43:52.201552 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.201451    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"default-dockercfg-bnmdp\""
Nov 03 20:43:52.211564 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.211522    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:52.248481 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.248447    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163cd80e-8575-45d0-89ff-6163f25c57e7-tls-certs\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.248677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.248500    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-dshm\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.248677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.248652    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-model-cache\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.248677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.248679    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6rk\" (UniqueName: \"kubernetes.io/projected/163cd80e-8575-45d0-89ff-6163f25c57e7-kube-api-access-kt6rk\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.248822 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.248788    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-home\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.248885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.248866    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-tmp-dir\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.349831 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.349786    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-home\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.350036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.349962    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-tmp-dir\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.350085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.350076    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163cd80e-8575-45d0-89ff-6163f25c57e7-tls-certs\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.350170 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.350136    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-dshm\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.350212 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.350203    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-model-cache\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.350251 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.350225    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6rk\" (UniqueName: \"kubernetes.io/projected/163cd80e-8575-45d0-89ff-6163f25c57e7-kube-api-access-kt6rk\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.350251 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.350230    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-home\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.350356 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.350276    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-tmp-dir\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.350482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.350436    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-model-cache\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.352515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.352490    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-dshm\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.352883 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.352860    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163cd80e-8575-45d0-89ff-6163f25c57e7-tls-certs\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.403860 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.403820    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6rk\" (UniqueName: \"kubernetes.io/projected/163cd80e-8575-45d0-89ff-6163f25c57e7-kube-api-access-kt6rk\") pod \"simulator-w2-b119f2-kserve-5d954846d-8jv8p\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") " pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.418760 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.418730    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:52.781257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.781161    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:52.794766 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.794738    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:52.799938 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:52.799908    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"]
Nov 03 20:43:52.805381 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:43:52.805354    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod163cd80e_8575_45d0_89ff_6163f25c57e7.slice/crio-85d1d7f73cbd1ad5a12da48df5496ecea11da2003166d9cffdf0d7a916a2acbb WatchSource:0}: Error finding container 85d1d7f73cbd1ad5a12da48df5496ecea11da2003166d9cffdf0d7a916a2acbb: Status 404 returned error can't find the container with id 85d1d7f73cbd1ad5a12da48df5496ecea11da2003166d9cffdf0d7a916a2acbb
Nov 03 20:43:53.495628 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:53.495594    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" event={"ID":"163cd80e-8575-45d0-89ff-6163f25c57e7","Type":"ContainerStarted","Data":"ebd740fce9cb9c0e5edd029257e144634d15d2b5dfde55bdd7b43b37ff855826"}
Nov 03 20:43:53.496120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:53.495638    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" event={"ID":"163cd80e-8575-45d0-89ff-6163f25c57e7","Type":"ContainerStarted","Data":"85d1d7f73cbd1ad5a12da48df5496ecea11da2003166d9cffdf0d7a916a2acbb"}
Nov 03 20:43:53.507639 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:53.507611    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:43:53.606079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:53.606011    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podStartSLOduration=1.605990587 podStartE2EDuration="1.605990587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:53.602965888 +0000 UTC m=+873.606924614" watchObservedRunningTime="2026-11-03 20:43:53.605990587 +0000 UTC m=+873.609949311"
Nov 03 20:43:53.752067 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:53.751972    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:53.921110 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:53.921060    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:54.284940 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:54.284903    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:54.419643 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:54.419604    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:43:59.065563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:43:59.065518    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:02.419898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:02.419860    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:44:02.433674 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:02.433643    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:44:02.569352 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:02.569313    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:44:03.847494 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:03.847454    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:08.562107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:08.562046    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:09.486560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:09.486519    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:44:13.272536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:13.272504    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:14.170433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:14.170398    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:17.027413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:17.027373    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:17.615731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:17.615694    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:17.809636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:17.809596    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:18.061119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:18.061014    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:18.485446 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:44:18.485403    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:44:20.225510 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:20.225485    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:44:20.231913 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:20.231885    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:44:20.232505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:20.232485    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:44:20.239476 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:20.239455    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:44:22.867651 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:22.867609    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:27.597586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:27.597551    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:28.870920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:28.870871    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:30.483994 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:30.483958    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:44:32.377247 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:32.377209    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:34.648301 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:34.648269    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:37.111478 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:37.111439    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:40.840647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:40.840607    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:41.898599 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:41.898562    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:44.658474 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:44.658433    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:46.675932 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:46.675892    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:48.484454 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:44:48.484415    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:44:51.447721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:51.447680    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:55.698337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:55.698297    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:44:56.203265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:44:56.203225    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:00.165406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.165358    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"]
Nov 03 20:45:00.169646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.169623    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"]
Nov 03 20:45:00.169794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.169771    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:00.174278 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.174255    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"]
Nov 03 20:45:00.174398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.174367    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:00.178216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.178195    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:00.211788 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.211758    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"]
Nov 03 20:45:00.215229 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.215204    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdf5\" (UniqueName: \"kubernetes.io/projected/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-kube-api-access-xtdf5\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg\" (UID: \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:00.215418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.215398    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg\" (UID: \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:00.216030 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.216005    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"]
Nov 03 20:45:00.216237 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.216152    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:00.220021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.220002    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"]
Nov 03 20:45:00.220186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.220166    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:00.226318 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.226294    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:00.226440 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.226386    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:45:00.262657 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.262622    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"]
Nov 03 20:45:00.266891 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.266864    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"]
Nov 03 20:45:00.306840 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.306804    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"]
Nov 03 20:45:00.310978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.310304    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"]
Nov 03 20:45:00.313739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.313706    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"]
Nov 03 20:45:00.316851 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.316819    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15b599f-4523-451a-8fe5-1fd4f1976df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq\" (UID: \"a15b599f-4523-451a-8fe5-1fd4f1976df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:00.317007 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.316886    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7e9f7-66a8-4352-a53d-9faac756296b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm\" (UID: \"f0e7e9f7-66a8-4352-a53d-9faac756296b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:00.317063 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317017    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg\" (UID: \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:00.317139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317075    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t\" (UID: \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:00.317215 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317198    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5hs\" (UniqueName: \"kubernetes.io/projected/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-kube-api-access-kj5hs\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t\" (UID: \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:00.317295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317277    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lz8l\" (UniqueName: \"kubernetes.io/projected/f0e7e9f7-66a8-4352-a53d-9faac756296b-kube-api-access-4lz8l\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm\" (UID: \"f0e7e9f7-66a8-4352-a53d-9faac756296b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:00.317349 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317315    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82222520-823e-452e-a946-11c6c91d52b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5\" (UID: \"82222520-823e-452e-a946-11c6c91d52b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:00.317460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317437    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011ee4ff-53ac-43a9-909b-d88411a76f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl\" (UID: \"011ee4ff-53ac-43a9-909b-d88411a76f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:00.317525 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317474    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zlcf\" (UniqueName: \"kubernetes.io/projected/011ee4ff-53ac-43a9-909b-d88411a76f86-kube-api-access-6zlcf\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl\" (UID: \"011ee4ff-53ac-43a9-909b-d88411a76f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:00.317525 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317513    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdf5\" (UniqueName: \"kubernetes.io/projected/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-kube-api-access-xtdf5\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg\" (UID: \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:00.317626 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317594    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdjfs\" (UniqueName: \"kubernetes.io/projected/82222520-823e-452e-a946-11c6c91d52b3-kube-api-access-gdjfs\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5\" (UID: \"82222520-823e-452e-a946-11c6c91d52b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:00.317683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317657    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsncj\" (UniqueName: \"kubernetes.io/projected/a15b599f-4523-451a-8fe5-1fd4f1976df1-kube-api-access-gsncj\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq\" (UID: \"a15b599f-4523-451a-8fe5-1fd4f1976df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:00.317905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.317878    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg\" (UID: \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:00.373218 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.373171    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"]
Nov 03 20:45:00.374715 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.374687    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdf5\" (UniqueName: \"kubernetes.io/projected/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-kube-api-access-xtdf5\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg\" (UID: \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:00.419043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.418957    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15b599f-4523-451a-8fe5-1fd4f1976df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq\" (UID: \"a15b599f-4523-451a-8fe5-1fd4f1976df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:00.419043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.418999    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7e9f7-66a8-4352-a53d-9faac756296b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm\" (UID: \"f0e7e9f7-66a8-4352-a53d-9faac756296b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:00.419316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419049    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t\" (UID: \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:00.419316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419080    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5hs\" (UniqueName: \"kubernetes.io/projected/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-kube-api-access-kj5hs\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t\" (UID: \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:00.419316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419143    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lz8l\" (UniqueName: \"kubernetes.io/projected/f0e7e9f7-66a8-4352-a53d-9faac756296b-kube-api-access-4lz8l\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm\" (UID: \"f0e7e9f7-66a8-4352-a53d-9faac756296b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:00.419316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419273    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82222520-823e-452e-a946-11c6c91d52b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5\" (UID: \"82222520-823e-452e-a946-11c6c91d52b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:00.419556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419345    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011ee4ff-53ac-43a9-909b-d88411a76f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl\" (UID: \"011ee4ff-53ac-43a9-909b-d88411a76f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:00.419556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419374    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zlcf\" (UniqueName: \"kubernetes.io/projected/011ee4ff-53ac-43a9-909b-d88411a76f86-kube-api-access-6zlcf\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl\" (UID: \"011ee4ff-53ac-43a9-909b-d88411a76f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:00.419556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419441    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdjfs\" (UniqueName: \"kubernetes.io/projected/82222520-823e-452e-a946-11c6c91d52b3-kube-api-access-gdjfs\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5\" (UID: \"82222520-823e-452e-a946-11c6c91d52b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:00.419556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419504    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsncj\" (UniqueName: \"kubernetes.io/projected/a15b599f-4523-451a-8fe5-1fd4f1976df1-kube-api-access-gsncj\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq\" (UID: \"a15b599f-4523-451a-8fe5-1fd4f1976df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:00.419941 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419917    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15b599f-4523-451a-8fe5-1fd4f1976df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq\" (UID: \"a15b599f-4523-451a-8fe5-1fd4f1976df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:00.419998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419917    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t\" (UID: \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:00.419998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419929    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7e9f7-66a8-4352-a53d-9faac756296b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm\" (UID: \"f0e7e9f7-66a8-4352-a53d-9faac756296b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:00.419998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.419965    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82222520-823e-452e-a946-11c6c91d52b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5\" (UID: \"82222520-823e-452e-a946-11c6c91d52b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:00.420285 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.420265    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011ee4ff-53ac-43a9-909b-d88411a76f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl\" (UID: \"011ee4ff-53ac-43a9-909b-d88411a76f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:00.474590 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.474553    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsncj\" (UniqueName: \"kubernetes.io/projected/a15b599f-4523-451a-8fe5-1fd4f1976df1-kube-api-access-gsncj\") pod \"maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq\" (UID: \"a15b599f-4523-451a-8fe5-1fd4f1976df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:00.474845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.474574    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lz8l\" (UniqueName: \"kubernetes.io/projected/f0e7e9f7-66a8-4352-a53d-9faac756296b-kube-api-access-4lz8l\") pod \"maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm\" (UID: \"f0e7e9f7-66a8-4352-a53d-9faac756296b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:00.474845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.474614    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5hs\" (UniqueName: \"kubernetes.io/projected/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-kube-api-access-kj5hs\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t\" (UID: \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:00.475027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.474866    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zlcf\" (UniqueName: \"kubernetes.io/projected/011ee4ff-53ac-43a9-909b-d88411a76f86-kube-api-access-6zlcf\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl\" (UID: \"011ee4ff-53ac-43a9-909b-d88411a76f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:00.475027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.474966    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdjfs\" (UniqueName: \"kubernetes.io/projected/82222520-823e-452e-a946-11c6c91d52b3-kube-api-access-gdjfs\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5\" (UID: \"82222520-823e-452e-a946-11c6c91d52b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:00.485652 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.485624    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:00.493306 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.493274    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:00.500154 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.500126    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:00.545115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.542234    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:00.553124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.552810    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:00.569314 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.564837    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:00.883645 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.883612    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"]
Nov 03 20:45:00.890308 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:00.890279    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9de338_b8ac_48e7_9b62_8ad5814c1d3d.slice/crio-176dfec28e8898ffeba8b3087d036f80352d0ce125aaba2bf3843a5f67690f4e WatchSource:0}: Error finding container 176dfec28e8898ffeba8b3087d036f80352d0ce125aaba2bf3843a5f67690f4e: Status 404 returned error can't find the container with id 176dfec28e8898ffeba8b3087d036f80352d0ce125aaba2bf3843a5f67690f4e
Nov 03 20:45:00.892429 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.892407    5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:45:00.952009 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.951922    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"]
Nov 03 20:45:00.955487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.955461    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"]
Nov 03 20:45:00.957307 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:00.957279    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e7e9f7_66a8_4352_a53d_9faac756296b.slice/crio-9221ffbcfa9fee31d498b629d44b99f010dab86ab53e59a58365e47df9a8971a WatchSource:0}: Error finding container 9221ffbcfa9fee31d498b629d44b99f010dab86ab53e59a58365e47df9a8971a: Status 404 returned error can't find the container with id 9221ffbcfa9fee31d498b629d44b99f010dab86ab53e59a58365e47df9a8971a
Nov 03 20:45:00.957945 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:00.957919    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15b599f_4523_451a_8fe5_1fd4f1976df1.slice/crio-6bfa74ae97fb70f505bf825f1113b45348028b8fc63974613a8bf0089b4425a2 WatchSource:0}: Error finding container 6bfa74ae97fb70f505bf825f1113b45348028b8fc63974613a8bf0089b4425a2: Status 404 returned error can't find the container with id 6bfa74ae97fb70f505bf825f1113b45348028b8fc63974613a8bf0089b4425a2
Nov 03 20:45:00.971018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.970983    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq" event={"ID":"a15b599f-4523-451a-8fe5-1fd4f1976df1","Type":"ContainerStarted","Data":"6bfa74ae97fb70f505bf825f1113b45348028b8fc63974613a8bf0089b4425a2"}
Nov 03 20:45:00.972441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.972412    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" event={"ID":"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d","Type":"ContainerStarted","Data":"176dfec28e8898ffeba8b3087d036f80352d0ce125aaba2bf3843a5f67690f4e"}
Nov 03 20:45:00.973863 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:00.973832    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" event={"ID":"f0e7e9f7-66a8-4352-a53d-9faac756296b","Type":"ContainerStarted","Data":"9221ffbcfa9fee31d498b629d44b99f010dab86ab53e59a58365e47df9a8971a"}
Nov 03 20:45:01.007727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:01.007681    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:01.012471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:01.012429    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"]
Nov 03 20:45:01.017588 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:01.017554    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011ee4ff_53ac_43a9_909b_d88411a76f86.slice/crio-3bd463e9336fd5c4ac9d1d420df5f767fdf374bfefa278ff0a773261f29171f4 WatchSource:0}: Error finding container 3bd463e9336fd5c4ac9d1d420df5f767fdf374bfefa278ff0a773261f29171f4: Status 404 returned error can't find the container with id 3bd463e9336fd5c4ac9d1d420df5f767fdf374bfefa278ff0a773261f29171f4
Nov 03 20:45:01.033483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:01.033452    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"]
Nov 03 20:45:01.039186 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:01.039147    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abb07b3_69af_47e3_a39d_0ccd9feba2ab.slice/crio-559091bdd2706c7864d3d4c8bfaaaa4bdeb6e1df5bb11d93b5aca35b608ea771 WatchSource:0}: Error finding container 559091bdd2706c7864d3d4c8bfaaaa4bdeb6e1df5bb11d93b5aca35b608ea771: Status 404 returned error can't find the container with id 559091bdd2706c7864d3d4c8bfaaaa4bdeb6e1df5bb11d93b5aca35b608ea771
Nov 03 20:45:01.040134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:01.040105    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"]
Nov 03 20:45:01.045055 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:01.045031    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82222520_823e_452e_a946_11c6c91d52b3.slice/crio-836102e64c23a609dfd91a80333a6e0a906908ce362ec001bbf364083a066f83 WatchSource:0}: Error finding container 836102e64c23a609dfd91a80333a6e0a906908ce362ec001bbf364083a066f83: Status 404 returned error can't find the container with id 836102e64c23a609dfd91a80333a6e0a906908ce362ec001bbf364083a066f83
Nov 03 20:45:01.982238 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:01.982198    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" event={"ID":"7abb07b3-69af-47e3-a39d-0ccd9feba2ab","Type":"ContainerStarted","Data":"559091bdd2706c7864d3d4c8bfaaaa4bdeb6e1df5bb11d93b5aca35b608ea771"}
Nov 03 20:45:01.983795 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:01.983769    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" event={"ID":"82222520-823e-452e-a946-11c6c91d52b3","Type":"ContainerStarted","Data":"836102e64c23a609dfd91a80333a6e0a906908ce362ec001bbf364083a066f83"}
Nov 03 20:45:01.985140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:01.985116    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" event={"ID":"011ee4ff-53ac-43a9-909b-d88411a76f86","Type":"ContainerStarted","Data":"3bd463e9336fd5c4ac9d1d420df5f767fdf374bfefa278ff0a773261f29171f4"}
Nov 03 20:45:05.961781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:05.961692    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:08.052637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.052605    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" event={"ID":"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d","Type":"ContainerStarted","Data":"e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003"}
Nov 03 20:45:08.053143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.052907    5196 scope.go:122] "RemoveContainer" containerID="e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003"
Nov 03 20:45:08.054461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.054435    5196 generic.go:356] "Generic (PLEG): container finished" podID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerID="45366c8820e9f48556a96b58bb491aea85cdb13658ff28b0942da178b27e7c48" exitCode=60
Nov 03 20:45:08.054573 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.054499    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" event={"ID":"f0e7e9f7-66a8-4352-a53d-9faac756296b","Type":"ContainerDied","Data":"45366c8820e9f48556a96b58bb491aea85cdb13658ff28b0942da178b27e7c48"}
Nov 03 20:45:08.054793 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.054777    5196 scope.go:122] "RemoveContainer" containerID="45366c8820e9f48556a96b58bb491aea85cdb13658ff28b0942da178b27e7c48"
Nov 03 20:45:08.056367 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.056340    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" event={"ID":"7abb07b3-69af-47e3-a39d-0ccd9feba2ab","Type":"ContainerStarted","Data":"7ba26b5c0ca3cc7da5a66d52cdf067ff161bf0570b5aa8c390813e5d09346c4f"}
Nov 03 20:45:08.058856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.058831    5196 generic.go:356] "Generic (PLEG): container finished" podID="82222520-823e-452e-a946-11c6c91d52b3" containerID="5c0c04bc862d03a14bd6b05dfb8c728c995694cd822806c5f96aeb9040b476c2" exitCode=60
Nov 03 20:45:08.058969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.058939    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" event={"ID":"82222520-823e-452e-a946-11c6c91d52b3","Type":"ContainerDied","Data":"5c0c04bc862d03a14bd6b05dfb8c728c995694cd822806c5f96aeb9040b476c2"}
Nov 03 20:45:08.059200 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.059145    5196 scope.go:122] "RemoveContainer" containerID="5c0c04bc862d03a14bd6b05dfb8c728c995694cd822806c5f96aeb9040b476c2"
Nov 03 20:45:08.060863 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.060807    5196 generic.go:356] "Generic (PLEG): container finished" podID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerID="662c2e3edc8769faaf7230d24c3c804deab80f0630bdd96112025596347dd2e7" exitCode=60
Nov 03 20:45:08.060930 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.060868    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" event={"ID":"011ee4ff-53ac-43a9-909b-d88411a76f86","Type":"ContainerDied","Data":"662c2e3edc8769faaf7230d24c3c804deab80f0630bdd96112025596347dd2e7"}
Nov 03 20:45:08.061053 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.061038    5196 scope.go:122] "RemoveContainer" containerID="662c2e3edc8769faaf7230d24c3c804deab80f0630bdd96112025596347dd2e7"
Nov 03 20:45:08.584154 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:08.584071    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" podStartSLOduration=2.206863709 podStartE2EDuration="8.584055844s" totalImagesPullingTime="6.377192135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:08.581374997 +0000 UTC m=+948.585333722" watchObservedRunningTime="2026-11-03 20:45:08.584055844 +0000 UTC m=+948.588014620"
Nov 03 20:45:09.070312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.070273    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" event={"ID":"011ee4ff-53ac-43a9-909b-d88411a76f86","Type":"ContainerStarted","Data":"f9c686ff727d03ec6145255db8af58447f9e3ccfb40826e8b9c1820988f88dd9"}
Nov 03 20:45:09.072388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.072360    5196 generic.go:356] "Generic (PLEG): container finished" podID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerID="e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003" exitCode=60
Nov 03 20:45:09.072388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.072384    5196 generic.go:356] "Generic (PLEG): container finished" podID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerID="34949153d6c450bb9f4aa687c5a1918e95bac044d94f1ee90569126c3d7abb2a" exitCode=60
Nov 03 20:45:09.072575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.072442    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" event={"ID":"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d","Type":"ContainerDied","Data":"e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003"}
Nov 03 20:45:09.072575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.072499    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" event={"ID":"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d","Type":"ContainerDied","Data":"34949153d6c450bb9f4aa687c5a1918e95bac044d94f1ee90569126c3d7abb2a"}
Nov 03 20:45:09.072575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.072531    5196 scope.go:122] "RemoveContainer" containerID="e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003"
Nov 03 20:45:09.072739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.072724    5196 scope.go:122] "RemoveContainer" containerID="34949153d6c450bb9f4aa687c5a1918e95bac044d94f1ee90569126c3d7abb2a"
Nov 03 20:45:09.073037 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:09.073015    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg_odh-ai-gateway-infra(7e9de338-b8ac-48e7-9b62-8ad5814c1d3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d"
Nov 03 20:45:09.074740 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.074718    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" event={"ID":"f0e7e9f7-66a8-4352-a53d-9faac756296b","Type":"ContainerStarted","Data":"300808ced85adf3bbbe06718256de0b9a903788261df236475fae3fe8a2a7200"}
Nov 03 20:45:09.076412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.076390    5196 generic.go:356] "Generic (PLEG): container finished" podID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerID="7ba26b5c0ca3cc7da5a66d52cdf067ff161bf0570b5aa8c390813e5d09346c4f" exitCode=60
Nov 03 20:45:09.076536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.076456    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" event={"ID":"7abb07b3-69af-47e3-a39d-0ccd9feba2ab","Type":"ContainerDied","Data":"7ba26b5c0ca3cc7da5a66d52cdf067ff161bf0570b5aa8c390813e5d09346c4f"}
Nov 03 20:45:09.076725 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.076708    5196 scope.go:122] "RemoveContainer" containerID="7ba26b5c0ca3cc7da5a66d52cdf067ff161bf0570b5aa8c390813e5d09346c4f"
Nov 03 20:45:09.078575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.078553    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" event={"ID":"82222520-823e-452e-a946-11c6c91d52b3","Type":"ContainerStarted","Data":"2b294eccd1834d11445000c17d91a86b7caebae53e521628e7c2ade74208b4de"}
Nov 03 20:45:09.090231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.090210    5196 scope.go:122] "RemoveContainer" containerID="e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003"
Nov 03 20:45:09.090577 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:09.090555    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003\": container with ID starting with e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003 not found: ID does not exist" containerID="e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003"
Nov 03 20:45:09.090641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.090587    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003"} err="failed to get container status \"e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003\": rpc error: code = NotFound desc = could not find container \"e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003\": container with ID starting with e2a95e7030423c356b0adc3dc19d91823b0bccee016f70eeda56b313165f6003 not found: ID does not exist"
Nov 03 20:45:09.177868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.177809    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" podStartSLOduration=3.292181169 podStartE2EDuration="9.177794851s" totalImagesPullingTime="5.885613682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:09.175579458 +0000 UTC m=+949.179538182" watchObservedRunningTime="2026-11-03 20:45:09.177794851 +0000 UTC m=+949.181753663"
Nov 03 20:45:09.282333 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.282281    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" podStartSLOduration=3.522829235 podStartE2EDuration="9.28226538s" totalImagesPullingTime="5.759436145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:09.281647508 +0000 UTC m=+949.285606251" watchObservedRunningTime="2026-11-03 20:45:09.28226538 +0000 UTC m=+949.286224100"
Nov 03 20:45:09.389014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:09.388950    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" podStartSLOduration=3.728452505 podStartE2EDuration="9.388937011s" totalImagesPullingTime="5.660484506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:09.385715944 +0000 UTC m=+949.389674682" watchObservedRunningTime="2026-11-03 20:45:09.388937011 +0000 UTC m=+949.392895761"
Nov 03 20:45:10.094615 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.094579    5196 generic.go:356] "Generic (PLEG): container finished" podID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerID="f9c686ff727d03ec6145255db8af58447f9e3ccfb40826e8b9c1820988f88dd9" exitCode=60
Nov 03 20:45:10.095069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.094661    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" event={"ID":"011ee4ff-53ac-43a9-909b-d88411a76f86","Type":"ContainerDied","Data":"f9c686ff727d03ec6145255db8af58447f9e3ccfb40826e8b9c1820988f88dd9"}
Nov 03 20:45:10.095069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.094767    5196 scope.go:122] "RemoveContainer" containerID="662c2e3edc8769faaf7230d24c3c804deab80f0630bdd96112025596347dd2e7"
Nov 03 20:45:10.095069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.094929    5196 scope.go:122] "RemoveContainer" containerID="f9c686ff727d03ec6145255db8af58447f9e3ccfb40826e8b9c1820988f88dd9"
Nov 03 20:45:10.095284 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:10.095258    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl_odh-ai-gateway-infra(011ee4ff-53ac-43a9-909b-d88411a76f86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86"
Nov 03 20:45:10.097154 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.097135    5196 scope.go:122] "RemoveContainer" containerID="34949153d6c450bb9f4aa687c5a1918e95bac044d94f1ee90569126c3d7abb2a"
Nov 03 20:45:10.097426 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:10.097408    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg_odh-ai-gateway-infra(7e9de338-b8ac-48e7-9b62-8ad5814c1d3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d"
Nov 03 20:45:10.098913 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.098886    5196 generic.go:356] "Generic (PLEG): container finished" podID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerID="300808ced85adf3bbbe06718256de0b9a903788261df236475fae3fe8a2a7200" exitCode=60
Nov 03 20:45:10.099024 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.098961    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" event={"ID":"f0e7e9f7-66a8-4352-a53d-9faac756296b","Type":"ContainerDied","Data":"300808ced85adf3bbbe06718256de0b9a903788261df236475fae3fe8a2a7200"}
Nov 03 20:45:10.099181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.099171    5196 scope.go:122] "RemoveContainer" containerID="300808ced85adf3bbbe06718256de0b9a903788261df236475fae3fe8a2a7200"
Nov 03 20:45:10.099415 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:10.099397    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm_odh-ai-gateway-infra(f0e7e9f7-66a8-4352-a53d-9faac756296b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b"
Nov 03 20:45:10.101067 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.101043    5196 generic.go:356] "Generic (PLEG): container finished" podID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerID="2f747ccdb02070dc6d2317e15dab626eecc2bbac1f3735bed9972e913dd86adb" exitCode=60
Nov 03 20:45:10.101464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.101446    5196 scope.go:122] "RemoveContainer" containerID="2f747ccdb02070dc6d2317e15dab626eecc2bbac1f3735bed9972e913dd86adb"
Nov 03 20:45:10.101731 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:10.101706    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t_odh-ai-gateway-infra(7abb07b3-69af-47e3-a39d-0ccd9feba2ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab"
Nov 03 20:45:10.103676 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.103644    5196 generic.go:356] "Generic (PLEG): container finished" podID="82222520-823e-452e-a946-11c6c91d52b3" containerID="2b294eccd1834d11445000c17d91a86b7caebae53e521628e7c2ade74208b4de" exitCode=60
Nov 03 20:45:10.104021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.104001    5196 scope.go:122] "RemoveContainer" containerID="2b294eccd1834d11445000c17d91a86b7caebae53e521628e7c2ade74208b4de"
Nov 03 20:45:10.104338 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:10.104317    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5_odh-ai-gateway-infra(82222520-823e-452e-a946-11c6c91d52b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" podUID="82222520-823e-452e-a946-11c6c91d52b3"
Nov 03 20:45:10.109896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.109869    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" event={"ID":"7abb07b3-69af-47e3-a39d-0ccd9feba2ab","Type":"ContainerDied","Data":"2f747ccdb02070dc6d2317e15dab626eecc2bbac1f3735bed9972e913dd86adb"}
Nov 03 20:45:10.110029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.109922    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" event={"ID":"82222520-823e-452e-a946-11c6c91d52b3","Type":"ContainerDied","Data":"2b294eccd1834d11445000c17d91a86b7caebae53e521628e7c2ade74208b4de"}
Nov 03 20:45:10.110168 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.110078    5196 scope.go:122] "RemoveContainer" containerID="45366c8820e9f48556a96b58bb491aea85cdb13658ff28b0942da178b27e7c48"
Nov 03 20:45:10.124249 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.124224    5196 scope.go:122] "RemoveContainer" containerID="7ba26b5c0ca3cc7da5a66d52cdf067ff161bf0570b5aa8c390813e5d09346c4f"
Nov 03 20:45:10.138660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.138634    5196 scope.go:122] "RemoveContainer" containerID="5c0c04bc862d03a14bd6b05dfb8c728c995694cd822806c5f96aeb9040b476c2"
Nov 03 20:45:10.758353 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:10.758310    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:11.113624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:11.113595    5196 scope.go:122] "RemoveContainer" containerID="2f747ccdb02070dc6d2317e15dab626eecc2bbac1f3735bed9972e913dd86adb"
Nov 03 20:45:11.114065 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:11.113914    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t_odh-ai-gateway-infra(7abb07b3-69af-47e3-a39d-0ccd9feba2ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab"
Nov 03 20:45:11.115320 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:11.115300    5196 scope.go:122] "RemoveContainer" containerID="2b294eccd1834d11445000c17d91a86b7caebae53e521628e7c2ade74208b4de"
Nov 03 20:45:11.115585 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:11.115557    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5_odh-ai-gateway-infra(82222520-823e-452e-a946-11c6c91d52b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" podUID="82222520-823e-452e-a946-11c6c91d52b3"
Nov 03 20:45:11.116910 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:11.116888    5196 scope.go:122] "RemoveContainer" containerID="f9c686ff727d03ec6145255db8af58447f9e3ccfb40826e8b9c1820988f88dd9"
Nov 03 20:45:11.117185 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:11.117165    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl_odh-ai-gateway-infra(011ee4ff-53ac-43a9-909b-d88411a76f86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86"
Nov 03 20:45:11.118655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:11.118639    5196 scope.go:122] "RemoveContainer" containerID="300808ced85adf3bbbe06718256de0b9a903788261df236475fae3fe8a2a7200"
Nov 03 20:45:11.118850 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:11.118834    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm_odh-ai-gateway-infra(f0e7e9f7-66a8-4352-a53d-9faac756296b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b"
Nov 03 20:45:13.554087 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:13.554014    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:15.045635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:15.045600    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:45:15.568639 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:15.568607    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:16.160548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:16.160510    5196 generic.go:356] "Generic (PLEG): container finished" podID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerID="c7517402c83221852a352120d178abf0b0e122e8a7104282483d6477818e5b52" exitCode=60
Nov 03 20:45:16.160988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:16.160576    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq" event={"ID":"a15b599f-4523-451a-8fe5-1fd4f1976df1","Type":"ContainerDied","Data":"c7517402c83221852a352120d178abf0b0e122e8a7104282483d6477818e5b52"}
Nov 03 20:45:16.160988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:16.160847    5196 scope.go:122] "RemoveContainer" containerID="c7517402c83221852a352120d178abf0b0e122e8a7104282483d6477818e5b52"
Nov 03 20:45:17.170752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:17.170709    5196 generic.go:356] "Generic (PLEG): container finished" podID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerID="fe0491e06b6f9aec5a1c37643aedfdf887f3a4a6e423c50ce62faa9526553939" exitCode=60
Nov 03 20:45:17.171240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:17.170783    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq" event={"ID":"a15b599f-4523-451a-8fe5-1fd4f1976df1","Type":"ContainerDied","Data":"fe0491e06b6f9aec5a1c37643aedfdf887f3a4a6e423c50ce62faa9526553939"}
Nov 03 20:45:17.171240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:17.170839    5196 scope.go:122] "RemoveContainer" containerID="c7517402c83221852a352120d178abf0b0e122e8a7104282483d6477818e5b52"
Nov 03 20:45:17.171240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:17.171113    5196 scope.go:122] "RemoveContainer" containerID="fe0491e06b6f9aec5a1c37643aedfdf887f3a4a6e423c50ce62faa9526553939"
Nov 03 20:45:17.171462 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:17.171443    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq_odh-ai-gateway-infra(a15b599f-4523-451a-8fe5-1fd4f1976df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1"
Nov 03 20:45:18.180385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:18.180356    5196 scope.go:122] "RemoveContainer" containerID="fe0491e06b6f9aec5a1c37643aedfdf887f3a4a6e423c50ce62faa9526553939"
Nov 03 20:45:18.180803 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:18.180631    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq_odh-ai-gateway-infra(a15b599f-4523-451a-8fe5-1fd4f1976df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1"
Nov 03 20:45:18.487079 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:18.486962    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:45:20.108230 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:20.108197    5196 scope.go:122] "RemoveContainer" containerID="2b294eccd1834d11445000c17d91a86b7caebae53e521628e7c2ade74208b4de"
Nov 03 20:45:20.275308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:20.275271    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:21.101871 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:21.101832    5196 scope.go:122] "RemoveContainer" containerID="300808ced85adf3bbbe06718256de0b9a903788261df236475fae3fe8a2a7200"
Nov 03 20:45:21.202979 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:21.202950    5196 generic.go:356] "Generic (PLEG): container finished" podID="82222520-823e-452e-a946-11c6c91d52b3" containerID="46b242303f2480da0461bac718f3b48fb675259abba637fc62dd06b8a9c324e1" exitCode=60
Nov 03 20:45:21.203442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:21.203024    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" event={"ID":"82222520-823e-452e-a946-11c6c91d52b3","Type":"ContainerDied","Data":"46b242303f2480da0461bac718f3b48fb675259abba637fc62dd06b8a9c324e1"}
Nov 03 20:45:21.203442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:21.203086    5196 scope.go:122] "RemoveContainer" containerID="2b294eccd1834d11445000c17d91a86b7caebae53e521628e7c2ade74208b4de"
Nov 03 20:45:21.203599 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:21.203474    5196 scope.go:122] "RemoveContainer" containerID="46b242303f2480da0461bac718f3b48fb675259abba637fc62dd06b8a9c324e1"
Nov 03 20:45:21.203763 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:21.203744    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5_odh-ai-gateway-infra(82222520-823e-452e-a946-11c6c91d52b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" podUID="82222520-823e-452e-a946-11c6c91d52b3"
Nov 03 20:45:22.212163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.212129    5196 generic.go:356] "Generic (PLEG): container finished" podID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerID="f892201f2c55a9c0356592b02a8bcc994c688ee1b7280991ec539047a7827090" exitCode=60
Nov 03 20:45:22.212660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.212203    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" event={"ID":"f0e7e9f7-66a8-4352-a53d-9faac756296b","Type":"ContainerDied","Data":"f892201f2c55a9c0356592b02a8bcc994c688ee1b7280991ec539047a7827090"}
Nov 03 20:45:22.212660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.212272    5196 scope.go:122] "RemoveContainer" containerID="300808ced85adf3bbbe06718256de0b9a903788261df236475fae3fe8a2a7200"
Nov 03 20:45:22.212660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.212656    5196 scope.go:122] "RemoveContainer" containerID="f892201f2c55a9c0356592b02a8bcc994c688ee1b7280991ec539047a7827090"
Nov 03 20:45:22.212970 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:22.212946    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm_odh-ai-gateway-infra(f0e7e9f7-66a8-4352-a53d-9faac756296b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b"
Nov 03 20:45:22.321857 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.321824    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"]
Nov 03 20:45:22.495442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.495415    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:22.507735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.507704    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82222520-823e-452e-a946-11c6c91d52b3-kube-api-access-gdjfs\" (UniqueName: \"kubernetes.io/projected/82222520-823e-452e-a946-11c6c91d52b3-kube-api-access-gdjfs\") pod \"82222520-823e-452e-a946-11c6c91d52b3\" (UID: \"82222520-823e-452e-a946-11c6c91d52b3\") "
Nov 03 20:45:22.507935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.507851    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82222520-823e-452e-a946-11c6c91d52b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82222520-823e-452e-a946-11c6c91d52b3-maas-api-service-ca\") pod \"82222520-823e-452e-a946-11c6c91d52b3\" (UID: \"82222520-823e-452e-a946-11c6c91d52b3\") "
Nov 03 20:45:22.508183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.508152    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82222520-823e-452e-a946-11c6c91d52b3-maas-api-service-ca" pod "82222520-823e-452e-a946-11c6c91d52b3" (UID: "82222520-823e-452e-a946-11c6c91d52b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:22.508330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.508315    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82222520-823e-452e-a946-11c6c91d52b3-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:22.510150 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.510124    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82222520-823e-452e-a946-11c6c91d52b3-kube-api-access-gdjfs" pod "82222520-823e-452e-a946-11c6c91d52b3" (UID: "82222520-823e-452e-a946-11c6c91d52b3"). InnerVolumeSpecName "kube-api-access-gdjfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:22.609904 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:22.609810    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdjfs\" (UniqueName: \"kubernetes.io/projected/82222520-823e-452e-a946-11c6c91d52b3-kube-api-access-gdjfs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:23.101310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.101262    5196 scope.go:122] "RemoveContainer" containerID="2f747ccdb02070dc6d2317e15dab626eecc2bbac1f3735bed9972e913dd86adb"
Nov 03 20:45:23.101580 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.101414    5196 scope.go:122] "RemoveContainer" containerID="f9c686ff727d03ec6145255db8af58447f9e3ccfb40826e8b9c1820988f88dd9"
Nov 03 20:45:23.223181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.223142    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"
Nov 03 20:45:23.223635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.223178    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5" event={"ID":"82222520-823e-452e-a946-11c6c91d52b3","Type":"ContainerDied","Data":"836102e64c23a609dfd91a80333a6e0a906908ce362ec001bbf364083a066f83"}
Nov 03 20:45:23.223635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.223241    5196 scope.go:122] "RemoveContainer" containerID="46b242303f2480da0461bac718f3b48fb675259abba637fc62dd06b8a9c324e1"
Nov 03 20:45:23.335770 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.335736    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"]
Nov 03 20:45:23.384849 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.384770    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"]
Nov 03 20:45:23.388276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.388246    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895645-gldn5"]
Nov 03 20:45:23.495877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.495846    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:23.520208 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.520177    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e7e9f7-66a8-4352-a53d-9faac756296b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7e9f7-66a8-4352-a53d-9faac756296b-maas-api-service-ca\") pod \"f0e7e9f7-66a8-4352-a53d-9faac756296b\" (UID: \"f0e7e9f7-66a8-4352-a53d-9faac756296b\") "
Nov 03 20:45:23.520432 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.520411    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e7e9f7-66a8-4352-a53d-9faac756296b-kube-api-access-4lz8l\" (UniqueName: \"kubernetes.io/projected/f0e7e9f7-66a8-4352-a53d-9faac756296b-kube-api-access-4lz8l\") pod \"f0e7e9f7-66a8-4352-a53d-9faac756296b\" (UID: \"f0e7e9f7-66a8-4352-a53d-9faac756296b\") "
Nov 03 20:45:23.520602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.520582    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e7e9f7-66a8-4352-a53d-9faac756296b-maas-api-service-ca" pod "f0e7e9f7-66a8-4352-a53d-9faac756296b" (UID: "f0e7e9f7-66a8-4352-a53d-9faac756296b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:23.520815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.520800    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7e9f7-66a8-4352-a53d-9faac756296b-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:23.522819 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.522792    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e7e9f7-66a8-4352-a53d-9faac756296b-kube-api-access-4lz8l" pod "f0e7e9f7-66a8-4352-a53d-9faac756296b" (UID: "f0e7e9f7-66a8-4352-a53d-9faac756296b"). InnerVolumeSpecName "kube-api-access-4lz8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:23.622236 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:23.622204    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lz8l\" (UniqueName: \"kubernetes.io/projected/f0e7e9f7-66a8-4352-a53d-9faac756296b-kube-api-access-4lz8l\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:24.102006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.101975    5196 scope.go:122] "RemoveContainer" containerID="34949153d6c450bb9f4aa687c5a1918e95bac044d94f1ee90569126c3d7abb2a"
Nov 03 20:45:24.111860 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.111826    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82222520-823e-452e-a946-11c6c91d52b3" path="/var/lib/kubelet/pods/82222520-823e-452e-a946-11c6c91d52b3/volumes"
Nov 03 20:45:24.234778 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.234737    5196 generic.go:356] "Generic (PLEG): container finished" podID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerID="9c62ae1eff8f0702460558fdbc991f132bd5e5cb4140c377c3f7f9db375fa255" exitCode=60
Nov 03 20:45:24.235293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.234820    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" event={"ID":"7abb07b3-69af-47e3-a39d-0ccd9feba2ab","Type":"ContainerDied","Data":"9c62ae1eff8f0702460558fdbc991f132bd5e5cb4140c377c3f7f9db375fa255"}
Nov 03 20:45:24.235293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.234886    5196 scope.go:122] "RemoveContainer" containerID="2f747ccdb02070dc6d2317e15dab626eecc2bbac1f3735bed9972e913dd86adb"
Nov 03 20:45:24.235378 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.235305    5196 scope.go:122] "RemoveContainer" containerID="9c62ae1eff8f0702460558fdbc991f132bd5e5cb4140c377c3f7f9db375fa255"
Nov 03 20:45:24.235559 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:24.235540    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t_odh-ai-gateway-infra(7abb07b3-69af-47e3-a39d-0ccd9feba2ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab"
Nov 03 20:45:24.239663 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.239642    5196 generic.go:356] "Generic (PLEG): container finished" podID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerID="9f35d0e3ea511040c8b2db5b0b3fff30ca067cbc879ffcc2526921a6ba173505" exitCode=60
Nov 03 20:45:24.239783 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.239722    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" event={"ID":"011ee4ff-53ac-43a9-909b-d88411a76f86","Type":"ContainerDied","Data":"9f35d0e3ea511040c8b2db5b0b3fff30ca067cbc879ffcc2526921a6ba173505"}
Nov 03 20:45:24.240212 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.240191    5196 scope.go:122] "RemoveContainer" containerID="9f35d0e3ea511040c8b2db5b0b3fff30ca067cbc879ffcc2526921a6ba173505"
Nov 03 20:45:24.240547 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:24.240521    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl_odh-ai-gateway-infra(011ee4ff-53ac-43a9-909b-d88411a76f86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86"
Nov 03 20:45:24.241395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.241350    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm" event={"ID":"f0e7e9f7-66a8-4352-a53d-9faac756296b","Type":"ContainerDied","Data":"9221ffbcfa9fee31d498b629d44b99f010dab86ab53e59a58365e47df9a8971a"}
Nov 03 20:45:24.241395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.241379    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"
Nov 03 20:45:24.250667 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.250643    5196 scope.go:122] "RemoveContainer" containerID="f9c686ff727d03ec6145255db8af58447f9e3ccfb40826e8b9c1820988f88dd9"
Nov 03 20:45:24.266681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.266654    5196 scope.go:122] "RemoveContainer" containerID="f892201f2c55a9c0356592b02a8bcc994c688ee1b7280991ec539047a7827090"
Nov 03 20:45:24.557403 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.557352    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"]
Nov 03 20:45:24.560938 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:24.560905    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a19980-29895645-96bwm"]
Nov 03 20:45:25.102155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.102123    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:25.256380 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.256349    5196 generic.go:356] "Generic (PLEG): container finished" podID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerID="70c5988b1df1a2d47def81c4a0b5296e5861b854133e6556a8af85ed0b197d2e" exitCode=60
Nov 03 20:45:25.256796 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.256427    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" event={"ID":"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d","Type":"ContainerDied","Data":"70c5988b1df1a2d47def81c4a0b5296e5861b854133e6556a8af85ed0b197d2e"}
Nov 03 20:45:25.256796 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.256489    5196 scope.go:122] "RemoveContainer" containerID="34949153d6c450bb9f4aa687c5a1918e95bac044d94f1ee90569126c3d7abb2a"
Nov 03 20:45:25.256902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.256854    5196 scope.go:122] "RemoveContainer" containerID="70c5988b1df1a2d47def81c4a0b5296e5861b854133e6556a8af85ed0b197d2e"
Nov 03 20:45:25.257181 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:25.257151    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg_odh-ai-gateway-infra(7e9de338-b8ac-48e7-9b62-8ad5814c1d3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d"
Nov 03 20:45:25.353758 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.353479    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"]
Nov 03 20:45:25.464315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.464281    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"]
Nov 03 20:45:25.525231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.525206    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:25.642749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.642716    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:25.651751 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.651721    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-kube-api-access-kj5hs\" (UniqueName: \"kubernetes.io/projected/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-kube-api-access-kj5hs\") pod \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\" (UID: \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\") "
Nov 03 20:45:25.651899 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.651821    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-maas-api-service-ca\") pod \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\" (UID: \"7abb07b3-69af-47e3-a39d-0ccd9feba2ab\") "
Nov 03 20:45:25.652186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.652158    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-maas-api-service-ca" pod "7abb07b3-69af-47e3-a39d-0ccd9feba2ab" (UID: "7abb07b3-69af-47e3-a39d-0ccd9feba2ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:25.654004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.653979    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-kube-api-access-kj5hs" pod "7abb07b3-69af-47e3-a39d-0ccd9feba2ab" (UID: "7abb07b3-69af-47e3-a39d-0ccd9feba2ab"). InnerVolumeSpecName "kube-api-access-kj5hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:25.752802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.752772    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011ee4ff-53ac-43a9-909b-d88411a76f86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011ee4ff-53ac-43a9-909b-d88411a76f86-maas-api-service-ca\") pod \"011ee4ff-53ac-43a9-909b-d88411a76f86\" (UID: \"011ee4ff-53ac-43a9-909b-d88411a76f86\") "
Nov 03 20:45:25.752802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.752804    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011ee4ff-53ac-43a9-909b-d88411a76f86-kube-api-access-6zlcf\" (UniqueName: \"kubernetes.io/projected/011ee4ff-53ac-43a9-909b-d88411a76f86-kube-api-access-6zlcf\") pod \"011ee4ff-53ac-43a9-909b-d88411a76f86\" (UID: \"011ee4ff-53ac-43a9-909b-d88411a76f86\") "
Nov 03 20:45:25.753162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.753138    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011ee4ff-53ac-43a9-909b-d88411a76f86-maas-api-service-ca" pod "011ee4ff-53ac-43a9-909b-d88411a76f86" (UID: "011ee4ff-53ac-43a9-909b-d88411a76f86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:25.753416 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.753397    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5hs\" (UniqueName: \"kubernetes.io/projected/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-kube-api-access-kj5hs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:25.753464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.753420    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/011ee4ff-53ac-43a9-909b-d88411a76f86-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:25.753464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.753432    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abb07b3-69af-47e3-a39d-0ccd9feba2ab-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:25.755055 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.755038    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011ee4ff-53ac-43a9-909b-d88411a76f86-kube-api-access-6zlcf" pod "011ee4ff-53ac-43a9-909b-d88411a76f86" (UID: "011ee4ff-53ac-43a9-909b-d88411a76f86"). InnerVolumeSpecName "kube-api-access-6zlcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:25.854622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:25.854580    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zlcf\" (UniqueName: \"kubernetes.io/projected/011ee4ff-53ac-43a9-909b-d88411a76f86-kube-api-access-6zlcf\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:26.109380 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.109345    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" path="/var/lib/kubelet/pods/f0e7e9f7-66a8-4352-a53d-9faac756296b/volumes"
Nov 03 20:45:26.268043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.268015    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"
Nov 03 20:45:26.268521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.268045    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl" event={"ID":"011ee4ff-53ac-43a9-909b-d88411a76f86","Type":"ContainerDied","Data":"3bd463e9336fd5c4ac9d1d420df5f767fdf374bfefa278ff0a773261f29171f4"}
Nov 03 20:45:26.268521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.268132    5196 scope.go:122] "RemoveContainer" containerID="9f35d0e3ea511040c8b2db5b0b3fff30ca067cbc879ffcc2526921a6ba173505"
Nov 03 20:45:26.271655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.271626    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t" event={"ID":"7abb07b3-69af-47e3-a39d-0ccd9feba2ab","Type":"ContainerDied","Data":"559091bdd2706c7864d3d4c8bfaaaa4bdeb6e1df5bb11d93b5aca35b608ea771"}
Nov 03 20:45:26.271775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.271662    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"
Nov 03 20:45:26.279810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.279789    5196 scope.go:122] "RemoveContainer" containerID="9c62ae1eff8f0702460558fdbc991f132bd5e5cb4140c377c3f7f9db375fa255"
Nov 03 20:45:26.373948 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.373852    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"]
Nov 03 20:45:26.427812 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.427771    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"]
Nov 03 20:45:26.434064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.434029    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895645-kdjwl"]
Nov 03 20:45:26.541769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.541731    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"]
Nov 03 20:45:26.545277 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.545244    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895645-4d62t"]
Nov 03 20:45:26.553441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.553414    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:26.663489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.663399    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-maas-api-service-ca\") pod \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\" (UID: \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\") "
Nov 03 20:45:26.663489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.663472    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-kube-api-access-xtdf5\" (UniqueName: \"kubernetes.io/projected/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-kube-api-access-xtdf5\") pod \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\" (UID: \"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d\") "
Nov 03 20:45:26.663854 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.663825    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-maas-api-service-ca" pod "7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" (UID: "7e9de338-b8ac-48e7-9b62-8ad5814c1d3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:26.663982 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.663968    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:26.665786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.665752    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-kube-api-access-xtdf5" pod "7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" (UID: "7e9de338-b8ac-48e7-9b62-8ad5814c1d3d"). InnerVolumeSpecName "kube-api-access-xtdf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:26.764910 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:26.764874    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdf5\" (UniqueName: \"kubernetes.io/projected/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d-kube-api-access-xtdf5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:27.281842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:27.281819    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"
Nov 03 20:45:27.282292 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:27.281813    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg" event={"ID":"7e9de338-b8ac-48e7-9b62-8ad5814c1d3d","Type":"ContainerDied","Data":"176dfec28e8898ffeba8b3087d036f80352d0ce125aaba2bf3843a5f67690f4e"}
Nov 03 20:45:27.282292 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:27.281922    5196 scope.go:122] "RemoveContainer" containerID="70c5988b1df1a2d47def81c4a0b5296e5861b854133e6556a8af85ed0b197d2e"
Nov 03 20:45:27.438269 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:27.438224    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"]
Nov 03 20:45:27.485255 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:27.485215    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895645-6nlfg"]
Nov 03 20:45:28.051062 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:28.051026    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:28.101411 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:28.101369    5196 scope.go:122] "RemoveContainer" containerID="fe0491e06b6f9aec5a1c37643aedfdf887f3a4a6e423c50ce62faa9526553939"
Nov 03 20:45:28.110012 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:28.109987    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" path="/var/lib/kubelet/pods/011ee4ff-53ac-43a9-909b-d88411a76f86/volumes"
Nov 03 20:45:28.110435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:28.110421    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" path="/var/lib/kubelet/pods/7abb07b3-69af-47e3-a39d-0ccd9feba2ab/volumes"
Nov 03 20:45:28.110761 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:28.110750    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" path="/var/lib/kubelet/pods/7e9de338-b8ac-48e7-9b62-8ad5814c1d3d/volumes"
Nov 03 20:45:29.126109 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126064    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"]
Nov 03 20:45:29.126884 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126861    5196 cpu_manager.go:414] "containerMap: removing container" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.126975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126885    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.126975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126900    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.126975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126907    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.126975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126942    5196 cpu_manager.go:414] "containerMap: removing container" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.126975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126951    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.126975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126968    5196 cpu_manager.go:414] "containerMap: removing container" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.126975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126974    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126988    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.126996    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127005    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127012    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127035    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127041    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127055    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127062    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127077    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127084    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127114    5196 cpu_manager.go:414] "containerMap: removing container" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127121    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127145    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127152    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127163    5196 cpu_manager.go:414] "containerMap: removing container" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127169    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127321    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127332    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127343    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.127351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127352    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.128266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127365    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.128266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127373    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.128266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127381    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.128266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127391    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.128266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127401    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.128266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127408    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.128266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.127419    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.131203 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.131183    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.182759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.182727    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"]
Nov 03 20:45:29.195901 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.195814    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.196078 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.195957    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgds5\" (UniqueName: \"kubernetes.io/projected/99883928-fe8a-4d39-806a-b9da6db79547-kube-api-access-jgds5\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.196078 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.196015    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.225501 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.225472    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:29.296916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.296883    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.297122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.296996    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.297122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.297043    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgds5\" (UniqueName: \"kubernetes.io/projected/99883928-fe8a-4d39-806a-b9da6db79547-kube-api-access-jgds5\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.297622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.297598    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.299823 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.299799    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.301015 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.300992    5196 generic.go:356] "Generic (PLEG): container finished" podID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerID="159c383685a488b015e92738ea3c052799bb6f159e74ac0de909c10493f44d20" exitCode=60
Nov 03 20:45:29.301110 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.301027    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq" event={"ID":"a15b599f-4523-451a-8fe5-1fd4f1976df1","Type":"ContainerDied","Data":"159c383685a488b015e92738ea3c052799bb6f159e74ac0de909c10493f44d20"}
Nov 03 20:45:29.301110 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.301071    5196 scope.go:122] "RemoveContainer" containerID="fe0491e06b6f9aec5a1c37643aedfdf887f3a4a6e423c50ce62faa9526553939"
Nov 03 20:45:29.301461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.301440    5196 scope.go:122] "RemoveContainer" containerID="159c383685a488b015e92738ea3c052799bb6f159e74ac0de909c10493f44d20"
Nov 03 20:45:29.301712 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:29.301693    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq_odh-ai-gateway-infra(a15b599f-4523-451a-8fe5-1fd4f1976df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1"
Nov 03 20:45:29.350949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.350919    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgds5\" (UniqueName: \"kubernetes.io/projected/99883928-fe8a-4d39-806a-b9da6db79547-kube-api-access-jgds5\") pod \"maas-api-revoke-sub-84daddc3-bc59v\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.445455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.445422    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:29.766118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.766007    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"]
Nov 03 20:45:29.766893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.766865    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.766893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.766892    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.767120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.766929    5196 cpu_manager.go:414] "containerMap: removing container" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.767120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.766936    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.767120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.766955    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.767120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.766963    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abb07b3-69af-47e3-a39d-0ccd9feba2ab" containerName="cleanup"
Nov 03 20:45:29.767330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.767125    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9de338-b8ac-48e7-9b62-8ad5814c1d3d" containerName="cleanup"
Nov 03 20:45:29.767330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.767137    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e7e9f7-66a8-4352-a53d-9faac756296b" containerName="cleanup"
Nov 03 20:45:29.767330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.767154    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="011ee4ff-53ac-43a9-909b-d88411a76f86" containerName="cleanup"
Nov 03 20:45:29.767330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.767175    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="82222520-823e-452e-a946-11c6c91d52b3" containerName="cleanup"
Nov 03 20:45:29.771864 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.771842    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:29.822824 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.822791    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"]
Nov 03 20:45:29.827747 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.827716    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"]
Nov 03 20:45:29.858823 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.857306    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:29.903708 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.903676    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:29.903892 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.903802    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:29.903892 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:29.903841    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntdw\" (UniqueName: \"kubernetes.io/projected/35581491-a0a2-4e82-82bb-d3469692ee77-kube-api-access-gntdw\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:30.005143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.005074    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntdw\" (UniqueName: \"kubernetes.io/projected/35581491-a0a2-4e82-82bb-d3469692ee77-kube-api-access-gntdw\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:30.005321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.005232    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:30.005321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.005290    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:30.005920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.005896    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:30.007786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.007766    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:30.058952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.058922    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntdw\" (UniqueName: \"kubernetes.io/projected/35581491-a0a2-4e82-82bb-d3469692ee77-kube-api-access-gntdw\") pod \"maas-api-revoke-sub-983f4439-5nxwf\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:30.088400 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.088362    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:30.310629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.310537    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v" event={"ID":"99883928-fe8a-4d39-806a-b9da6db79547","Type":"ContainerStarted","Data":"a3dbb0f8536b043d150b0de9b9754a9556e4d0a8f02e856b2dcc35538b590291"}
Nov 03 20:45:30.310629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.310600    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v" event={"ID":"99883928-fe8a-4d39-806a-b9da6db79547","Type":"ContainerStarted","Data":"6b4bdbe0006ad6670f7d6f1ee82dc1d4e8be4651347291adedc6b7f7924e5485"}
Nov 03 20:45:30.418641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.418609    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"]
Nov 03 20:45:30.421156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.421108    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v" podStartSLOduration=1.421081265 podStartE2EDuration="1.421081265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:30.418196021 +0000 UTC m=+970.422154757" watchObservedRunningTime="2026-11-03 20:45:30.421081265 +0000 UTC m=+970.425040010"
Nov 03 20:45:30.468957 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.468923    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"]
Nov 03 20:45:30.525066 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:30.524740    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35581491_a0a2_4e82_82bb_d3469692ee77.slice/crio-1d2e47c5b365a6b0ae986d7b562049c8de8ee1050241923f1b74af28bd23af62 WatchSource:0}: Error finding container 1d2e47c5b365a6b0ae986d7b562049c8de8ee1050241923f1b74af28bd23af62: Status 404 returned error can't find the container with id 1d2e47c5b365a6b0ae986d7b562049c8de8ee1050241923f1b74af28bd23af62
Nov 03 20:45:30.596032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.596003    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:30.713067 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.713032    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15b599f-4523-451a-8fe5-1fd4f1976df1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15b599f-4523-451a-8fe5-1fd4f1976df1-maas-api-service-ca\") pod \"a15b599f-4523-451a-8fe5-1fd4f1976df1\" (UID: \"a15b599f-4523-451a-8fe5-1fd4f1976df1\") "
Nov 03 20:45:30.713067 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.713070    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15b599f-4523-451a-8fe5-1fd4f1976df1-kube-api-access-gsncj\" (UniqueName: \"kubernetes.io/projected/a15b599f-4523-451a-8fe5-1fd4f1976df1-kube-api-access-gsncj\") pod \"a15b599f-4523-451a-8fe5-1fd4f1976df1\" (UID: \"a15b599f-4523-451a-8fe5-1fd4f1976df1\") "
Nov 03 20:45:30.713490 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.713462    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15b599f-4523-451a-8fe5-1fd4f1976df1-maas-api-service-ca" pod "a15b599f-4523-451a-8fe5-1fd4f1976df1" (UID: "a15b599f-4523-451a-8fe5-1fd4f1976df1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:30.713726 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.713706    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15b599f-4523-451a-8fe5-1fd4f1976df1-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:30.716048 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.716019    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15b599f-4523-451a-8fe5-1fd4f1976df1-kube-api-access-gsncj" pod "a15b599f-4523-451a-8fe5-1fd4f1976df1" (UID: "a15b599f-4523-451a-8fe5-1fd4f1976df1"). InnerVolumeSpecName "kube-api-access-gsncj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:30.815407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:30.815363    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsncj\" (UniqueName: \"kubernetes.io/projected/a15b599f-4523-451a-8fe5-1fd4f1976df1-kube-api-access-gsncj\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:31.321403 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.321352    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf" event={"ID":"35581491-a0a2-4e82-82bb-d3469692ee77","Type":"ContainerStarted","Data":"2a32da781c06f4ad795737054622f3c7caf2646bd7f85765d1f9d5c33f117a1c"}
Nov 03 20:45:31.321846 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.321423    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf" event={"ID":"35581491-a0a2-4e82-82bb-d3469692ee77","Type":"ContainerStarted","Data":"1d2e47c5b365a6b0ae986d7b562049c8de8ee1050241923f1b74af28bd23af62"}
Nov 03 20:45:31.323122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.323068    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"
Nov 03 20:45:31.323246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.323122    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq" event={"ID":"a15b599f-4523-451a-8fe5-1fd4f1976df1","Type":"ContainerDied","Data":"6bfa74ae97fb70f505bf825f1113b45348028b8fc63974613a8bf0089b4425a2"}
Nov 03 20:45:31.323246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.323180    5196 scope.go:122] "RemoveContainer" containerID="159c383685a488b015e92738ea3c052799bb6f159e74ac0de909c10493f44d20"
Nov 03 20:45:31.325351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.325323    5196 generic.go:356] "Generic (PLEG): container finished" podID="99883928-fe8a-4d39-806a-b9da6db79547" containerID="a3dbb0f8536b043d150b0de9b9754a9556e4d0a8f02e856b2dcc35538b590291" exitCode=0
Nov 03 20:45:31.325473 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.325407    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v" event={"ID":"99883928-fe8a-4d39-806a-b9da6db79547","Type":"ContainerDied","Data":"a3dbb0f8536b043d150b0de9b9754a9556e4d0a8f02e856b2dcc35538b590291"}
Nov 03 20:45:31.434389 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.434334    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf" podStartSLOduration=2.43431779 podStartE2EDuration="2.43431779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:31.431860969 +0000 UTC m=+971.435819721" watchObservedRunningTime="2026-11-03 20:45:31.43431779 +0000 UTC m=+971.438276513"
Nov 03 20:45:31.544156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.544119    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"]
Nov 03 20:45:31.547677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:31.547649    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fd9afe-29895645-ww2mq"]
Nov 03 20:45:32.109767 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.109729    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" path="/var/lib/kubelet/pods/a15b599f-4523-451a-8fe5-1fd4f1976df1/volumes"
Nov 03 20:45:32.336076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.336042    5196 generic.go:356] "Generic (PLEG): container finished" podID="35581491-a0a2-4e82-82bb-d3469692ee77" containerID="2a32da781c06f4ad795737054622f3c7caf2646bd7f85765d1f9d5c33f117a1c" exitCode=0
Nov 03 20:45:32.336511 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.336128    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf" event={"ID":"35581491-a0a2-4e82-82bb-d3469692ee77","Type":"ContainerDied","Data":"2a32da781c06f4ad795737054622f3c7caf2646bd7f85765d1f9d5c33f117a1c"}
Nov 03 20:45:32.511376 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.511351    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:32.636377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.636346    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-openshift-ca\") pod \"99883928-fe8a-4d39-806a-b9da6db79547\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") "
Nov 03 20:45:32.636569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.636431    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-xks-ca\") pod \"99883928-fe8a-4d39-806a-b9da6db79547\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") "
Nov 03 20:45:32.636569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.636481    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99883928-fe8a-4d39-806a-b9da6db79547-kube-api-access-jgds5\" (UniqueName: \"kubernetes.io/projected/99883928-fe8a-4d39-806a-b9da6db79547-kube-api-access-jgds5\") pod \"99883928-fe8a-4d39-806a-b9da6db79547\" (UID: \"99883928-fe8a-4d39-806a-b9da6db79547\") "
Nov 03 20:45:32.636726 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.636701    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-openshift-ca" pod "99883928-fe8a-4d39-806a-b9da6db79547" (UID: "99883928-fe8a-4d39-806a-b9da6db79547"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:32.637024 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.636998    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:32.638804 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.638772    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-xks-ca" pod "99883928-fe8a-4d39-806a-b9da6db79547" (UID: "99883928-fe8a-4d39-806a-b9da6db79547"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:32.638939 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.638843    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99883928-fe8a-4d39-806a-b9da6db79547-kube-api-access-jgds5" pod "99883928-fe8a-4d39-806a-b9da6db79547" (UID: "99883928-fe8a-4d39-806a-b9da6db79547"). InnerVolumeSpecName "kube-api-access-jgds5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:32.738522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.738491    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99883928-fe8a-4d39-806a-b9da6db79547-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:32.738522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:32.738520    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgds5\" (UniqueName: \"kubernetes.io/projected/99883928-fe8a-4d39-806a-b9da6db79547-kube-api-access-jgds5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:33.346771 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.346735    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v" event={"ID":"99883928-fe8a-4d39-806a-b9da6db79547","Type":"ContainerDied","Data":"6b4bdbe0006ad6670f7d6f1ee82dc1d4e8be4651347291adedc6b7f7924e5485"}
Nov 03 20:45:33.346771 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.346767    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"
Nov 03 20:45:33.347298 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.346795    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4bdbe0006ad6670f7d6f1ee82dc1d4e8be4651347291adedc6b7f7924e5485"
Nov 03 20:45:33.503739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.503715    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:33.648463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.648371    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-xks-ca\") pod \"35581491-a0a2-4e82-82bb-d3469692ee77\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") "
Nov 03 20:45:33.648463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.648419    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-openshift-ca\") pod \"35581491-a0a2-4e82-82bb-d3469692ee77\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") "
Nov 03 20:45:33.648728 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.648509    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35581491-a0a2-4e82-82bb-d3469692ee77-kube-api-access-gntdw\" (UniqueName: \"kubernetes.io/projected/35581491-a0a2-4e82-82bb-d3469692ee77-kube-api-access-gntdw\") pod \"35581491-a0a2-4e82-82bb-d3469692ee77\" (UID: \"35581491-a0a2-4e82-82bb-d3469692ee77\") "
Nov 03 20:45:33.648911 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.648880    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-openshift-ca" pod "35581491-a0a2-4e82-82bb-d3469692ee77" (UID: "35581491-a0a2-4e82-82bb-d3469692ee77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:33.650769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.650743    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-xks-ca" pod "35581491-a0a2-4e82-82bb-d3469692ee77" (UID: "35581491-a0a2-4e82-82bb-d3469692ee77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:33.650895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.650805    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35581491-a0a2-4e82-82bb-d3469692ee77-kube-api-access-gntdw" pod "35581491-a0a2-4e82-82bb-d3469692ee77" (UID: "35581491-a0a2-4e82-82bb-d3469692ee77"). InnerVolumeSpecName "kube-api-access-gntdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:33.749738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.749697    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntdw\" (UniqueName: \"kubernetes.io/projected/35581491-a0a2-4e82-82bb-d3469692ee77-kube-api-access-gntdw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:33.749738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.749738    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:33.749738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.749747    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35581491-a0a2-4e82-82bb-d3469692ee77-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:33.860308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:33.860263    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:45:34.365119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:34.364486    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"
Nov 03 20:45:34.365119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:34.364518    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf" event={"ID":"35581491-a0a2-4e82-82bb-d3469692ee77","Type":"ContainerDied","Data":"1d2e47c5b365a6b0ae986d7b562049c8de8ee1050241923f1b74af28bd23af62"}
Nov 03 20:45:34.365119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:34.364610    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d2e47c5b365a6b0ae986d7b562049c8de8ee1050241923f1b74af28bd23af62"
Nov 03 20:45:34.600283 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:34.600237    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:39.394497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:39.394445    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:41.282772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:41.282733    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:44.364661 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:44.364621    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:45.186449 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.186416    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"]
Nov 03 20:45:45.187015 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187002    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:45.187069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187017    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:45.187069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187040    5196 cpu_manager.go:414] "containerMap: removing container" podUID="99883928-fe8a-4d39-806a-b9da6db79547" containerName="revoke-keys"
Nov 03 20:45:45.187069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187046    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99883928-fe8a-4d39-806a-b9da6db79547" containerName="revoke-keys"
Nov 03 20:45:45.187069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187069    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:45.187307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187075    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:45.187307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187106    5196 cpu_manager.go:414] "containerMap: removing container" podUID="35581491-a0a2-4e82-82bb-d3469692ee77" containerName="revoke-keys"
Nov 03 20:45:45.187307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187111    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35581491-a0a2-4e82-82bb-d3469692ee77" containerName="revoke-keys"
Nov 03 20:45:45.187307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187197    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:45.187307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187203    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:45.187307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187212    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="35581491-a0a2-4e82-82bb-d3469692ee77" containerName="revoke-keys"
Nov 03 20:45:45.187307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187219    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:45.187307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.187228    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="99883928-fe8a-4d39-806a-b9da6db79547" containerName="revoke-keys"
Nov 03 20:45:45.189705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.189678    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:45.194355 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.194336    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.243774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.243735    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"]
Nov 03 20:45:45.281797 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.281757    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.282010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.281942    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.282010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.281987    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgbc\" (UniqueName: \"kubernetes.io/projected/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-kube-api-access-gtgbc\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.286139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.286114    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:45.286272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.286214    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:45:45.383458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.383417    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.383921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.383564    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.383921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.383603    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgbc\" (UniqueName: \"kubernetes.io/projected/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-kube-api-access-gtgbc\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.384134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.384108    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.386149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.386126    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.438459 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.438366    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgbc\" (UniqueName: \"kubernetes.io/projected/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-kube-api-access-gtgbc\") pod \"maas-api-revoke-sub-fffb86cd-c6n8q\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.509063 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.509032    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:45.887769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:45.887737    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"]
Nov 03 20:45:45.892949 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:45.892916    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc616e8e7_02ee_4fe8_89b2_dc1bd2e23482.slice/crio-d5548d6b18de01767efc7796377a431b17833207d36a03b1a53f23eb068ae18f WatchSource:0}: Error finding container d5548d6b18de01767efc7796377a431b17833207d36a03b1a53f23eb068ae18f: Status 404 returned error can't find the container with id d5548d6b18de01767efc7796377a431b17833207d36a03b1a53f23eb068ae18f
Nov 03 20:45:46.459202 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:46.459120    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q" event={"ID":"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482","Type":"ContainerStarted","Data":"16f1fd1ba6ed2e7432d561d0f6c24b1f4ec653bfdfb65f3810a27d0eb2035e02"}
Nov 03 20:45:46.459202 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:46.459172    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q" event={"ID":"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482","Type":"ContainerStarted","Data":"d5548d6b18de01767efc7796377a431b17833207d36a03b1a53f23eb068ae18f"}
Nov 03 20:45:46.566819 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:46.566765    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q" podStartSLOduration=1.566751899 podStartE2EDuration="1.566751899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:46.565256072 +0000 UTC m=+986.569214822" watchObservedRunningTime="2026-11-03 20:45:46.566751899 +0000 UTC m=+986.570710622"
Nov 03 20:45:47.477748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:47.477711    5196 generic.go:356] "Generic (PLEG): container finished" podID="c616e8e7-02ee-4fe8-89b2-dc1bd2e23482" containerID="16f1fd1ba6ed2e7432d561d0f6c24b1f4ec653bfdfb65f3810a27d0eb2035e02" exitCode=0
Nov 03 20:45:47.478197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:47.477802    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q" event={"ID":"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482","Type":"ContainerDied","Data":"16f1fd1ba6ed2e7432d561d0f6c24b1f4ec653bfdfb65f3810a27d0eb2035e02"}
Nov 03 20:45:48.489245 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:45:48.489201    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:45:48.692321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.692290    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:48.736053 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.736013    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-openshift-ca\") pod \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") "
Nov 03 20:45:48.736264 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.736121    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-kube-api-access-gtgbc\" (UniqueName: \"kubernetes.io/projected/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-kube-api-access-gtgbc\") pod \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") "
Nov 03 20:45:48.736264 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.736232    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-xks-ca\") pod \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\" (UID: \"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482\") "
Nov 03 20:45:48.736498 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.736449    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-openshift-ca" pod "c616e8e7-02ee-4fe8-89b2-dc1bd2e23482" (UID: "c616e8e7-02ee-4fe8-89b2-dc1bd2e23482"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:48.736629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.736607    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:48.739143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.739047    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-kube-api-access-gtgbc" pod "c616e8e7-02ee-4fe8-89b2-dc1bd2e23482" (UID: "c616e8e7-02ee-4fe8-89b2-dc1bd2e23482"). InnerVolumeSpecName "kube-api-access-gtgbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:48.739271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.739144    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-xks-ca" pod "c616e8e7-02ee-4fe8-89b2-dc1bd2e23482" (UID: "c616e8e7-02ee-4fe8-89b2-dc1bd2e23482"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:48.837900 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.837854    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtgbc\" (UniqueName: \"kubernetes.io/projected/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-kube-api-access-gtgbc\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:48.837900 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:48.837900    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:49.195163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:49.195123    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:49.496424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:49.496329    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"
Nov 03 20:45:49.496424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:49.496340    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q" event={"ID":"c616e8e7-02ee-4fe8-89b2-dc1bd2e23482","Type":"ContainerDied","Data":"d5548d6b18de01767efc7796377a431b17833207d36a03b1a53f23eb068ae18f"}
Nov 03 20:45:49.496424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:49.496390    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5548d6b18de01767efc7796377a431b17833207d36a03b1a53f23eb068ae18f"
Nov 03 20:45:53.990617 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:53.990527    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:54.520535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.520495    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"]
Nov 03 20:45:54.521145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.521126    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:54.521145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.521141    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15b599f-4523-451a-8fe5-1fd4f1976df1" containerName="cleanup"
Nov 03 20:45:54.521302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.521177    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c616e8e7-02ee-4fe8-89b2-dc1bd2e23482" containerName="revoke-keys"
Nov 03 20:45:54.521302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.521183    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c616e8e7-02ee-4fe8-89b2-dc1bd2e23482" containerName="revoke-keys"
Nov 03 20:45:54.521302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.521274    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c616e8e7-02ee-4fe8-89b2-dc1bd2e23482" containerName="revoke-keys"
Nov 03 20:45:54.524814 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.524798    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.578457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.578420    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"]
Nov 03 20:45:54.605919 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.605883    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.606107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.605925    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.606168 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.606076    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sm6f\" (UniqueName: \"kubernetes.io/projected/25841a65-a68a-450b-b6f2-1b54b45e8908-kube-api-access-4sm6f\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.621235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.621197    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:54.621235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.621208    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:45:54.707668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.707633    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sm6f\" (UniqueName: \"kubernetes.io/projected/25841a65-a68a-450b-b6f2-1b54b45e8908-kube-api-access-4sm6f\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.707869 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.707767    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.707869 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.707795    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.708593 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.708572    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.710552 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.710525    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.762128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.762059    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sm6f\" (UniqueName: \"kubernetes.io/projected/25841a65-a68a-450b-b6f2-1b54b45e8908-kube-api-access-4sm6f\") pod \"maas-api-revoke-sub-14ea385e-rhw6v\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.838722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.838685    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:54.878072 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:54.878029    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:55.214976 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:55.214888    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"]
Nov 03 20:45:55.220139 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:45:55.220087    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25841a65_a68a_450b_b6f2_1b54b45e8908.slice/crio-b111996b209af01ececfa77727587bbcfe3044dd2718ad7d038ce85bf0addcb5 WatchSource:0}: Error finding container b111996b209af01ececfa77727587bbcfe3044dd2718ad7d038ce85bf0addcb5: Status 404 returned error can't find the container with id b111996b209af01ececfa77727587bbcfe3044dd2718ad7d038ce85bf0addcb5
Nov 03 20:45:55.543598 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:55.543565    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v" event={"ID":"25841a65-a68a-450b-b6f2-1b54b45e8908","Type":"ContainerStarted","Data":"b111996b209af01ececfa77727587bbcfe3044dd2718ad7d038ce85bf0addcb5"}
Nov 03 20:45:56.551631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:56.551538    5196 generic.go:356] "Generic (PLEG): container finished" podID="25841a65-a68a-450b-b6f2-1b54b45e8908" containerID="3cc43c4c6c9dfa18991c304ab54cd3e3549d2d130693609975425a68249d1f12" exitCode=0
Nov 03 20:45:56.552038 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:56.551625    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v" event={"ID":"25841a65-a68a-450b-b6f2-1b54b45e8908","Type":"ContainerDied","Data":"3cc43c4c6c9dfa18991c304ab54cd3e3549d2d130693609975425a68249d1f12"}
Nov 03 20:45:57.721537 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.721511    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:57.846350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.846318    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-xks-ca\") pod \"25841a65-a68a-450b-b6f2-1b54b45e8908\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") "
Nov 03 20:45:57.846549 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.846367    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25841a65-a68a-450b-b6f2-1b54b45e8908-kube-api-access-4sm6f\" (UniqueName: \"kubernetes.io/projected/25841a65-a68a-450b-b6f2-1b54b45e8908-kube-api-access-4sm6f\") pod \"25841a65-a68a-450b-b6f2-1b54b45e8908\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") "
Nov 03 20:45:57.846549 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.846422    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-openshift-ca\") pod \"25841a65-a68a-450b-b6f2-1b54b45e8908\" (UID: \"25841a65-a68a-450b-b6f2-1b54b45e8908\") "
Nov 03 20:45:57.846892 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.846857    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-openshift-ca" pod "25841a65-a68a-450b-b6f2-1b54b45e8908" (UID: "25841a65-a68a-450b-b6f2-1b54b45e8908"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:57.847024 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.846978    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:57.848728 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.848696    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25841a65-a68a-450b-b6f2-1b54b45e8908-kube-api-access-4sm6f" pod "25841a65-a68a-450b-b6f2-1b54b45e8908" (UID: "25841a65-a68a-450b-b6f2-1b54b45e8908"). InnerVolumeSpecName "kube-api-access-4sm6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:57.848848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.848760    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-xks-ca" pod "25841a65-a68a-450b-b6f2-1b54b45e8908" (UID: "25841a65-a68a-450b-b6f2-1b54b45e8908"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:57.947981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.947949    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25841a65-a68a-450b-b6f2-1b54b45e8908-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:57.947981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:57.947978    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sm6f\" (UniqueName: \"kubernetes.io/projected/25841a65-a68a-450b-b6f2-1b54b45e8908-kube-api-access-4sm6f\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:58.567327 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:58.567246    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"
Nov 03 20:45:58.567327 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:58.567284    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v" event={"ID":"25841a65-a68a-450b-b6f2-1b54b45e8908","Type":"ContainerDied","Data":"b111996b209af01ececfa77727587bbcfe3044dd2718ad7d038ce85bf0addcb5"}
Nov 03 20:45:58.567517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:58.567343    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b111996b209af01ececfa77727587bbcfe3044dd2718ad7d038ce85bf0addcb5"
Nov 03 20:45:58.764852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:45:58.764814    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:03.658675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:03.658627    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:05.197003 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.196960    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"]
Nov 03 20:46:05.197658 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.197642    5196 cpu_manager.go:414] "containerMap: removing container" podUID="25841a65-a68a-450b-b6f2-1b54b45e8908" containerName="revoke-keys"
Nov 03 20:46:05.197701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.197659    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25841a65-a68a-450b-b6f2-1b54b45e8908" containerName="revoke-keys"
Nov 03 20:46:05.197779 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.197770    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="25841a65-a68a-450b-b6f2-1b54b45e8908" containerName="revoke-keys"
Nov 03 20:46:05.201574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.201553    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.220368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.220327    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"]
Nov 03 20:46:05.226407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.226380    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.255683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.255646    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"]
Nov 03 20:46:05.259134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.259069    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:46:05.259593 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.259576    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:46:05.298601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.298575    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"default-dockercfg-9fwld\""
Nov 03 20:46:05.298947 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.298926    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"premium-w4-984cbc-kserve-self-signed-certs\""
Nov 03 20:46:05.299139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.298957    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"kube-root-ca.crt\""
Nov 03 20:46:05.299139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.298964    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-984cbc\"/\"openshift-service-ca.crt\""
Nov 03 20:46:05.304536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.304509    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"]
Nov 03 20:46:05.331565 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.331534    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jj6\" (UniqueName: \"kubernetes.io/projected/6249b701-4a12-439f-b47d-287004b120b8-kube-api-access-t4jj6\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.331731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.331600    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-dshm\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.331731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.331645    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-model-cache\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.331861 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.331833    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6249b701-4a12-439f-b47d-287004b120b8-tls-certs\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.331917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.331870    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.331959 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.331928    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.332011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.331983    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fns4\" (UniqueName: \"kubernetes.io/projected/653a316f-049d-47e8-a03e-a36ed0869755-kube-api-access-7fns4\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.332120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.332074    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-tmp-dir\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.332197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.332178    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-home\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.332375 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.332357    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:05.433490 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433457    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jj6\" (UniqueName: \"kubernetes.io/projected/6249b701-4a12-439f-b47d-287004b120b8-kube-api-access-t4jj6\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.433680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433510    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-dshm\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.433680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433526    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-model-cache\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.433680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433581    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6249b701-4a12-439f-b47d-287004b120b8-tls-certs\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.433680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433604    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.433680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433633    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.433952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433687    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fns4\" (UniqueName: \"kubernetes.io/projected/653a316f-049d-47e8-a03e-a36ed0869755-kube-api-access-7fns4\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.433952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433732    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-tmp-dir\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.433952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.433779    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-home\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.434125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.434067    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-model-cache\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.434202 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.434144    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-home\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.434487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.434467    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-tmp-dir\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.434907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.434874    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.436213 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.436189    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-dshm\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.436394 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.436374    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6249b701-4a12-439f-b47d-287004b120b8-tls-certs\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.436644 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.436618    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.487631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.487559    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fns4\" (UniqueName: \"kubernetes.io/projected/653a316f-049d-47e8-a03e-a36ed0869755-kube-api-access-7fns4\") pod \"maas-api-revoke-sub-b3f97cb6-lx58q\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.487790 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.487699    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jj6\" (UniqueName: \"kubernetes.io/projected/6249b701-4a12-439f-b47d-287004b120b8-kube-api-access-t4jj6\") pod \"premium-w4-984cbc-kserve-7c79f9db7f-9b7ks\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") " pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.522359 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.522316    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:05.539898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.539868    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:05.915460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.915427    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"]
Nov 03 20:46:05.931340 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:46:05.931309    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653a316f_049d_47e8_a03e_a36ed0869755.slice/crio-5876c36e4266cf349803e35ca006d591c32ddbdc16f5bc5cbe2a8328802ce278 WatchSource:0}: Error finding container 5876c36e4266cf349803e35ca006d591c32ddbdc16f5bc5cbe2a8328802ce278: Status 404 returned error can't find the container with id 5876c36e4266cf349803e35ca006d591c32ddbdc16f5bc5cbe2a8328802ce278
Nov 03 20:46:05.966428 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:05.966400    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"]
Nov 03 20:46:05.970445 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:46:05.970416    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6249b701_4a12_439f_b47d_287004b120b8.slice/crio-e48b399b4b1087c6c29af4b1fed5454de2fc144e99fddb8e7bd49bd4cff4de76 WatchSource:0}: Error finding container e48b399b4b1087c6c29af4b1fed5454de2fc144e99fddb8e7bd49bd4cff4de76: Status 404 returned error can't find the container with id e48b399b4b1087c6c29af4b1fed5454de2fc144e99fddb8e7bd49bd4cff4de76
Nov 03 20:46:06.634505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:06.634311    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q" event={"ID":"653a316f-049d-47e8-a03e-a36ed0869755","Type":"ContainerStarted","Data":"66ffe0f9cab6fbccda6ca0feb96c2bedadf64b64e7be591bcd8db64b550a75fc"}
Nov 03 20:46:06.634505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:06.634387    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q" event={"ID":"653a316f-049d-47e8-a03e-a36ed0869755","Type":"ContainerStarted","Data":"5876c36e4266cf349803e35ca006d591c32ddbdc16f5bc5cbe2a8328802ce278"}
Nov 03 20:46:06.636884 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:06.636304    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" event={"ID":"6249b701-4a12-439f-b47d-287004b120b8","Type":"ContainerStarted","Data":"5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4"}
Nov 03 20:46:06.636884 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:06.636350    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" event={"ID":"6249b701-4a12-439f-b47d-287004b120b8","Type":"ContainerStarted","Data":"e48b399b4b1087c6c29af4b1fed5454de2fc144e99fddb8e7bd49bd4cff4de76"}
Nov 03 20:46:06.734294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:06.734255    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:46:06.742451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:06.742386    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q" podStartSLOduration=1.7423681389999999 podStartE2EDuration="1.742368139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:06.740968065 +0000 UTC m=+1006.744926800" watchObservedRunningTime="2026-11-03 20:46:06.742368139 +0000 UTC m=+1006.746326862"
Nov 03 20:46:06.856916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:06.853649    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podStartSLOduration=1.853628929 podStartE2EDuration="1.853628929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:06.853016705 +0000 UTC m=+1006.856975454" watchObservedRunningTime="2026-11-03 20:46:06.853628929 +0000 UTC m=+1006.857587654"
Nov 03 20:46:07.540226 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:07.540185    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:07.645878 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:07.645841    5196 generic.go:356] "Generic (PLEG): container finished" podID="653a316f-049d-47e8-a03e-a36ed0869755" containerID="66ffe0f9cab6fbccda6ca0feb96c2bedadf64b64e7be591bcd8db64b550a75fc" exitCode=0
Nov 03 20:46:07.646313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:07.645921    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q" event={"ID":"653a316f-049d-47e8-a03e-a36ed0869755","Type":"ContainerDied","Data":"66ffe0f9cab6fbccda6ca0feb96c2bedadf64b64e7be591bcd8db64b550a75fc"}
Nov 03 20:46:08.476468 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.476432    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:08.827470 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.827425    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:08.982924 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.982879    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-openshift-ca\") pod \"653a316f-049d-47e8-a03e-a36ed0869755\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") "
Nov 03 20:46:08.983135 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.982939    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653a316f-049d-47e8-a03e-a36ed0869755-kube-api-access-7fns4\" (UniqueName: \"kubernetes.io/projected/653a316f-049d-47e8-a03e-a36ed0869755-kube-api-access-7fns4\") pod \"653a316f-049d-47e8-a03e-a36ed0869755\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") "
Nov 03 20:46:08.983242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.983220    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-xks-ca\") pod \"653a316f-049d-47e8-a03e-a36ed0869755\" (UID: \"653a316f-049d-47e8-a03e-a36ed0869755\") "
Nov 03 20:46:08.983379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.983338    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-openshift-ca" pod "653a316f-049d-47e8-a03e-a36ed0869755" (UID: "653a316f-049d-47e8-a03e-a36ed0869755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:08.983766 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.983681    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:08.985888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.985851    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653a316f-049d-47e8-a03e-a36ed0869755-kube-api-access-7fns4" pod "653a316f-049d-47e8-a03e-a36ed0869755" (UID: "653a316f-049d-47e8-a03e-a36ed0869755"). InnerVolumeSpecName "kube-api-access-7fns4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:08.986141 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:08.986080    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-xks-ca" pod "653a316f-049d-47e8-a03e-a36ed0869755" (UID: "653a316f-049d-47e8-a03e-a36ed0869755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:09.084656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:09.084603    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fns4\" (UniqueName: \"kubernetes.io/projected/653a316f-049d-47e8-a03e-a36ed0869755-kube-api-access-7fns4\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:09.084656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:09.084642    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653a316f-049d-47e8-a03e-a36ed0869755-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:09.665858 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:09.665828    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"
Nov 03 20:46:09.665858 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:09.665845    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q" event={"ID":"653a316f-049d-47e8-a03e-a36ed0869755","Type":"ContainerDied","Data":"5876c36e4266cf349803e35ca006d591c32ddbdc16f5bc5cbe2a8328802ce278"}
Nov 03 20:46:09.666065 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:09.665901    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5876c36e4266cf349803e35ca006d591c32ddbdc16f5bc5cbe2a8328802ce278"
Nov 03 20:46:13.250186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:13.250153    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:15.540755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.540726    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:15.553922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.553886    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:15.721539 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.721506    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:46:15.942289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.942255    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"]
Nov 03 20:46:15.942951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.942930    5196 cpu_manager.go:414] "containerMap: removing container" podUID="653a316f-049d-47e8-a03e-a36ed0869755" containerName="revoke-keys"
Nov 03 20:46:15.943070 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.942954    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653a316f-049d-47e8-a03e-a36ed0869755" containerName="revoke-keys"
Nov 03 20:46:15.943206 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.943191    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="653a316f-049d-47e8-a03e-a36ed0869755" containerName="revoke-keys"
Nov 03 20:46:15.947035 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.947013    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:15.963137 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.963106    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ttw\" (UniqueName: \"kubernetes.io/projected/56d6b4ff-8175-454c-9b4a-0a86b435dea5-kube-api-access-r4ttw\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:15.963283 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.963156    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:15.963324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.963303    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:15.997165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.997131    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:46:15.997452 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:15.997422    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:46:16.001720 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.001696    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"]
Nov 03 20:46:16.064337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.064296    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ttw\" (UniqueName: \"kubernetes.io/projected/56d6b4ff-8175-454c-9b4a-0a86b435dea5-kube-api-access-r4ttw\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:16.064337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.064336    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:16.064603 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.064438    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:16.065031 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.065008    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:16.066990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.066966    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:16.118523 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.118491    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ttw\" (UniqueName: \"kubernetes.io/projected/56d6b4ff-8175-454c-9b4a-0a86b435dea5-kube-api-access-r4ttw\") pod \"maas-api-revoke-sub-7b18abc3-tn6s5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:16.261277 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.261198    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:16.654976 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.654948    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"]
Nov 03 20:46:16.658209 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:46:16.658173    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d6b4ff_8175_454c_9b4a_0a86b435dea5.slice/crio-155c53e310bdd116f53f6afd9d863056106935cc8a4ed9358122b7a0f35271f6 WatchSource:0}: Error finding container 155c53e310bdd116f53f6afd9d863056106935cc8a4ed9358122b7a0f35271f6: Status 404 returned error can't find the container with id 155c53e310bdd116f53f6afd9d863056106935cc8a4ed9358122b7a0f35271f6
Nov 03 20:46:16.718933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:16.718896    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5" event={"ID":"56d6b4ff-8175-454c-9b4a-0a86b435dea5","Type":"ContainerStarted","Data":"155c53e310bdd116f53f6afd9d863056106935cc8a4ed9358122b7a0f35271f6"}
Nov 03 20:46:17.727856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:17.727819    5196 generic.go:356] "Generic (PLEG): container finished" podID="56d6b4ff-8175-454c-9b4a-0a86b435dea5" containerID="eb9816f034f66da53a048fa6408a3f865ac5c890ad120e932aff377277d6a809" exitCode=0
Nov 03 20:46:17.728315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:17.727898    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5" event={"ID":"56d6b4ff-8175-454c-9b4a-0a86b435dea5","Type":"ContainerDied","Data":"eb9816f034f66da53a048fa6408a3f865ac5c890ad120e932aff377277d6a809"}
Nov 03 20:46:17.979477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:17.979442    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:18.486208 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:46:18.486162    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:46:18.912332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:18.912297    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:18.999862 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:18.999827    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d6b4ff-8175-454c-9b4a-0a86b435dea5-kube-api-access-r4ttw\" (UniqueName: \"kubernetes.io/projected/56d6b4ff-8175-454c-9b4a-0a86b435dea5-kube-api-access-r4ttw\") pod \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") "
Nov 03 20:46:19.000034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:18.999960    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-xks-ca\") pod \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") "
Nov 03 20:46:19.000034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:18.999992    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-openshift-ca\") pod \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\" (UID: \"56d6b4ff-8175-454c-9b4a-0a86b435dea5\") "
Nov 03 20:46:19.000397 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.000370    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-openshift-ca" pod "56d6b4ff-8175-454c-9b4a-0a86b435dea5" (UID: "56d6b4ff-8175-454c-9b4a-0a86b435dea5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:19.000502 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.000469    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:19.002217 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.002187    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d6b4ff-8175-454c-9b4a-0a86b435dea5-kube-api-access-r4ttw" pod "56d6b4ff-8175-454c-9b4a-0a86b435dea5" (UID: "56d6b4ff-8175-454c-9b4a-0a86b435dea5"). InnerVolumeSpecName "kube-api-access-r4ttw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:19.002560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.002533    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-xks-ca" pod "56d6b4ff-8175-454c-9b4a-0a86b435dea5" (UID: "56d6b4ff-8175-454c-9b4a-0a86b435dea5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:19.101607 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.101573    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d6b4ff-8175-454c-9b4a-0a86b435dea5-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:19.101607 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.101605    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4ttw\" (UniqueName: \"kubernetes.io/projected/56d6b4ff-8175-454c-9b4a-0a86b435dea5-kube-api-access-r4ttw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:19.752142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.749642    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"
Nov 03 20:46:19.752142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.749674    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5" event={"ID":"56d6b4ff-8175-454c-9b4a-0a86b435dea5","Type":"ContainerDied","Data":"155c53e310bdd116f53f6afd9d863056106935cc8a4ed9358122b7a0f35271f6"}
Nov 03 20:46:19.752142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:19.749725    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="155c53e310bdd116f53f6afd9d863056106935cc8a4ed9358122b7a0f35271f6"
Nov 03 20:46:22.681313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:22.681276    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:27.568360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:27.568318    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:32.556148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:32.556106    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:33.256405 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.256372    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"]
Nov 03 20:46:33.257014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.257000    5196 cpu_manager.go:414] "containerMap: removing container" podUID="56d6b4ff-8175-454c-9b4a-0a86b435dea5" containerName="revoke-keys"
Nov 03 20:46:33.257058 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.257016    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d6b4ff-8175-454c-9b4a-0a86b435dea5" containerName="revoke-keys"
Nov 03 20:46:33.257159 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.257149    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d6b4ff-8175-454c-9b4a-0a86b435dea5" containerName="revoke-keys"
Nov 03 20:46:33.260565 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.260550    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.320390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.320350    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"]
Nov 03 20:46:33.356677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.356649    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:46:33.356939 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.356922    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:46:33.375069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.375034    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjkb\" (UniqueName: \"kubernetes.io/projected/ef277745-615c-4e9e-9c98-7c1f2e7455cf-kube-api-access-lsjkb\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.375262 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.375219    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.375310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.375262    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.476850 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.476807    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.477066 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.476868    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.477066 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.476965    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjkb\" (UniqueName: \"kubernetes.io/projected/ef277745-615c-4e9e-9c98-7c1f2e7455cf-kube-api-access-lsjkb\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.477702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.477671    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.479456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.479433    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.531323 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.531243    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjkb\" (UniqueName: \"kubernetes.io/projected/ef277745-615c-4e9e-9c98-7c1f2e7455cf-kube-api-access-lsjkb\") pod \"maas-api-revoke-sub-211bcda9-db97d\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.574297 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.574262    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:33.953829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:33.953796    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"]
Nov 03 20:46:33.958676 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:46:33.958644    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef277745_615c_4e9e_9c98_7c1f2e7455cf.slice/crio-cda81125fd42e8061fd8263b608bb4b17ea72434f692bd20831df88a490857f8 WatchSource:0}: Error finding container cda81125fd42e8061fd8263b608bb4b17ea72434f692bd20831df88a490857f8: Status 404 returned error can't find the container with id cda81125fd42e8061fd8263b608bb4b17ea72434f692bd20831df88a490857f8
Nov 03 20:46:34.862447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:34.862416    5196 generic.go:356] "Generic (PLEG): container finished" podID="ef277745-615c-4e9e-9c98-7c1f2e7455cf" containerID="3b6db91887ec5b687d6d9514b3122d9211414aa7ed94971c3a51fc9081dc8bed" exitCode=0
Nov 03 20:46:34.862827 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:34.862505    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d" event={"ID":"ef277745-615c-4e9e-9c98-7c1f2e7455cf","Type":"ContainerDied","Data":"3b6db91887ec5b687d6d9514b3122d9211414aa7ed94971c3a51fc9081dc8bed"}
Nov 03 20:46:34.862827 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:34.862560    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d" event={"ID":"ef277745-615c-4e9e-9c98-7c1f2e7455cf","Type":"ContainerStarted","Data":"cda81125fd42e8061fd8263b608bb4b17ea72434f692bd20831df88a490857f8"}
Nov 03 20:46:36.046388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.046363    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:36.205968 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.205879    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-xks-ca\") pod \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") "
Nov 03 20:46:36.205968 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.205946    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef277745-615c-4e9e-9c98-7c1f2e7455cf-kube-api-access-lsjkb\" (UniqueName: \"kubernetes.io/projected/ef277745-615c-4e9e-9c98-7c1f2e7455cf-kube-api-access-lsjkb\") pod \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") "
Nov 03 20:46:36.206196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.206003    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-openshift-ca\") pod \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\" (UID: \"ef277745-615c-4e9e-9c98-7c1f2e7455cf\") "
Nov 03 20:46:36.206454 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.206426    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-openshift-ca" pod "ef277745-615c-4e9e-9c98-7c1f2e7455cf" (UID: "ef277745-615c-4e9e-9c98-7c1f2e7455cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:36.208294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.208270    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-xks-ca" pod "ef277745-615c-4e9e-9c98-7c1f2e7455cf" (UID: "ef277745-615c-4e9e-9c98-7c1f2e7455cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:36.208409 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.208383    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef277745-615c-4e9e-9c98-7c1f2e7455cf-kube-api-access-lsjkb" pod "ef277745-615c-4e9e-9c98-7c1f2e7455cf" (UID: "ef277745-615c-4e9e-9c98-7c1f2e7455cf"). InnerVolumeSpecName "kube-api-access-lsjkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:36.307690 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.307642    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.307690 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.307690    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsjkb\" (UniqueName: \"kubernetes.io/projected/ef277745-615c-4e9e-9c98-7c1f2e7455cf-kube-api-access-lsjkb\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.307941 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.307708    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef277745-615c-4e9e-9c98-7c1f2e7455cf-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.887373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.887335    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"
Nov 03 20:46:36.887373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.887367    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d" event={"ID":"ef277745-615c-4e9e-9c98-7c1f2e7455cf","Type":"ContainerDied","Data":"cda81125fd42e8061fd8263b608bb4b17ea72434f692bd20831df88a490857f8"}
Nov 03 20:46:36.887635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:36.887436    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda81125fd42e8061fd8263b608bb4b17ea72434f692bd20831df88a490857f8"
Nov 03 20:46:37.286473 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:37.286073    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:37.381040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:37.380997    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:46:39.342721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.342660    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"]
Nov 03 20:46:39.344483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.344449    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ef277745-615c-4e9e-9c98-7c1f2e7455cf" containerName="revoke-keys"
Nov 03 20:46:39.344662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.344644    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef277745-615c-4e9e-9c98-7c1f2e7455cf" containerName="revoke-keys"
Nov 03 20:46:39.345005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.344985    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef277745-615c-4e9e-9c98-7c1f2e7455cf" containerName="revoke-keys"
Nov 03 20:46:39.356440 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.356409    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.409147 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.408357    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fd9afe-openshift-default-dockercfg-k948v\""
Nov 03 20:46:39.506251 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506209    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506277    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdfe258-b7fb-41a0-b137-9740277878d5-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506339    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpqs\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-kube-api-access-qlpqs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506368    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506394    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506420    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506474    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506507    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506547    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.506893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.506829    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"]
Nov 03 20:46:39.607316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607216    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.607316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607270    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdfe258-b7fb-41a0-b137-9740277878d5-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.607316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607315    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpqs\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-kube-api-access-qlpqs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.607613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607333    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.607613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607350    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.607613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607368    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.607613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607401    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.607613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607423    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.607613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607451    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.608007 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.607978    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.608247 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.608218    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.608361 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.608252    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.608437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.608421    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.608578 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.608510    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdfe258-b7fb-41a0-b137-9740277878d5-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.610118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.610081    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.610503 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.610485    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.668284 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.668211    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.683005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.682914    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpqs\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-kube-api-access-qlpqs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:39.695993 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:39.695613    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:40.210448 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:40.210396    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"]
Nov 03 20:46:40.222795 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:40.222731    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:40.222932 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:40.222904    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:40.223044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:40.223022    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:40.962668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:40.959668    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" event={"ID":"5cdfe258-b7fb-41a0-b137-9740277878d5","Type":"ContainerStarted","Data":"ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c"}
Nov 03 20:46:40.962668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:40.959737    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" event={"ID":"5cdfe258-b7fb-41a0-b137-9740277878d5","Type":"ContainerStarted","Data":"ec1e5368712ef6ee35fac3604d713975390448f90142ab85a4d31d01046ff469"}
Nov 03 20:46:41.466686 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:41.466640    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:41.699530 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:41.699496    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:41.704264 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:41.704223    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:46:41.704518 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:41.704490    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 20:46:42.204656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:42.204618    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:42.696680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:42.696638    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:46:42.696895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:42.696704    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 20:46:43.697103 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:43.697069    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:46:43.697642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:43.697140    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 20:46:44.696691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:44.696648    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:46:44.696965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:44.696711    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 20:46:45.696408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:45.696366    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:46:45.696828 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:45.696427    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 20:46:46.697222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.697190    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:46:46.697759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.697247    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 20:46:46.828551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.828493    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podStartSLOduration=7.828476731 podStartE2EDuration="7.828476731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:41.073426149 +0000 UTC m=+1041.077384872" watchObservedRunningTime="2026-11-03 20:46:46.828476731 +0000 UTC m=+1046.832435454"
Nov 03 20:46:46.834075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.834035    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"]
Nov 03 20:46:46.848439 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.848405    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:46.878330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.878294    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bdr\" (UniqueName: \"kubernetes.io/projected/e454dd07-e27d-4498-ac05-53ccf1f341b6-kube-api-access-v4bdr\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:46.878542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.878419    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:46.878542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.878451    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:46.888681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.888644    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"]
Nov 03 20:46:46.898879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.898846    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:46:46.931524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.931487    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:46:46.980275 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.980186    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:46.980275 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.980254    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:46.980462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.980443    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bdr\" (UniqueName: \"kubernetes.io/projected/e454dd07-e27d-4498-ac05-53ccf1f341b6-kube-api-access-v4bdr\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:46.980969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.980937    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:46.982825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:46.982802    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:47.015910 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:47.015873    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:47.034396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:47.034361    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bdr\" (UniqueName: \"kubernetes.io/projected/e454dd07-e27d-4498-ac05-53ccf1f341b6-kube-api-access-v4bdr\") pod \"maas-api-revoke-sub-55802e18-v4lwt\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:47.166329 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:47.166298    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:47.553009 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:47.552902    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"]
Nov 03 20:46:47.559701 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:46:47.559670    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode454dd07_e27d_4498_ac05_53ccf1f341b6.slice/crio-2166996feea1584cff3ec7a32bc769057c738a9f165a8861bcadb328b415089b WatchSource:0}: Error finding container 2166996feea1584cff3ec7a32bc769057c738a9f165a8861bcadb328b415089b: Status 404 returned error can't find the container with id 2166996feea1584cff3ec7a32bc769057c738a9f165a8861bcadb328b415089b
Nov 03 20:46:47.700521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:47.700482    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:47.701004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:47.700869    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:47.701646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:47.701628    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 20:46:48.028268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:48.028223    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt" event={"ID":"e454dd07-e27d-4498-ac05-53ccf1f341b6","Type":"ContainerStarted","Data":"d36c7d128d74dbdf6bc2bcfc304d631c989c462a1ec51b61e775daff77f88740"}
Nov 03 20:46:48.028462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:48.028321    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt" event={"ID":"e454dd07-e27d-4498-ac05-53ccf1f341b6","Type":"ContainerStarted","Data":"2166996feea1584cff3ec7a32bc769057c738a9f165a8861bcadb328b415089b"}
Nov 03 20:46:48.140622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:48.140507    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt" podStartSLOduration=2.140492683 podStartE2EDuration="2.140492683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:48.137887872 +0000 UTC m=+1048.141846619" watchObservedRunningTime="2026-11-03 20:46:48.140492683 +0000 UTC m=+1048.144451407"
Nov 03 20:46:48.486861 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:46:48.486769    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:46:49.035963 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:49.035927    5196 generic.go:356] "Generic (PLEG): container finished" podID="e454dd07-e27d-4498-ac05-53ccf1f341b6" containerID="d36c7d128d74dbdf6bc2bcfc304d631c989c462a1ec51b61e775daff77f88740" exitCode=0
Nov 03 20:46:49.036500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:49.036010    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt" event={"ID":"e454dd07-e27d-4498-ac05-53ccf1f341b6","Type":"ContainerDied","Data":"d36c7d128d74dbdf6bc2bcfc304d631c989c462a1ec51b61e775daff77f88740"}
Nov 03 20:46:49.903047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:49.903017    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"]
Nov 03 20:46:49.907468 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:49.907447    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:49.960875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:49.960837    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"]
Nov 03 20:46:50.015816 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.015777    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.015998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.015900    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvqw\" (UniqueName: \"kubernetes.io/projected/859533d1-0dc9-40dc-8baf-d494518ba039-kube-api-access-fwvqw\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.015998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.015949    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.117493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.117462    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvqw\" (UniqueName: \"kubernetes.io/projected/859533d1-0dc9-40dc-8baf-d494518ba039-kube-api-access-fwvqw\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.117971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.117519    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.117971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.117676    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.118514 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.118450    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.120762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.120736    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.172307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.172233    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvqw\" (UniqueName: \"kubernetes.io/projected/859533d1-0dc9-40dc-8baf-d494518ba039-kube-api-access-fwvqw\") pod \"maas-api-revoke-sub-4ceebcd8-zbtb9\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.220294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.220268    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:50.221662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.221646    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:50.318953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.318920    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-xks-ca\") pod \"e454dd07-e27d-4498-ac05-53ccf1f341b6\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") "
Nov 03 20:46:50.319192 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.319059    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e454dd07-e27d-4498-ac05-53ccf1f341b6-kube-api-access-v4bdr\" (UniqueName: \"kubernetes.io/projected/e454dd07-e27d-4498-ac05-53ccf1f341b6-kube-api-access-v4bdr\") pod \"e454dd07-e27d-4498-ac05-53ccf1f341b6\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") "
Nov 03 20:46:50.319273 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.319250    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-openshift-ca\") pod \"e454dd07-e27d-4498-ac05-53ccf1f341b6\" (UID: \"e454dd07-e27d-4498-ac05-53ccf1f341b6\") "
Nov 03 20:46:50.319657 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.319623    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-openshift-ca" pod "e454dd07-e27d-4498-ac05-53ccf1f341b6" (UID: "e454dd07-e27d-4498-ac05-53ccf1f341b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:50.321751 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.321726    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-xks-ca" pod "e454dd07-e27d-4498-ac05-53ccf1f341b6" (UID: "e454dd07-e27d-4498-ac05-53ccf1f341b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:50.321991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.321971    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e454dd07-e27d-4498-ac05-53ccf1f341b6-kube-api-access-v4bdr" pod "e454dd07-e27d-4498-ac05-53ccf1f341b6" (UID: "e454dd07-e27d-4498-ac05-53ccf1f341b6"). InnerVolumeSpecName "kube-api-access-v4bdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:50.420493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.420454    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:50.420493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.420486    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4bdr\" (UniqueName: \"kubernetes.io/projected/e454dd07-e27d-4498-ac05-53ccf1f341b6-kube-api-access-v4bdr\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:50.420705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.420503    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e454dd07-e27d-4498-ac05-53ccf1f341b6-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:50.597291 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:50.597252    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"]
Nov 03 20:46:50.601672 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:46:50.601645    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859533d1_0dc9_40dc_8baf_d494518ba039.slice/crio-2a47934365aafd7cb60c14e53da60635ab36f56eb47f00f32591a2e7c3bceea4 WatchSource:0}: Error finding container 2a47934365aafd7cb60c14e53da60635ab36f56eb47f00f32591a2e7c3bceea4: Status 404 returned error can't find the container with id 2a47934365aafd7cb60c14e53da60635ab36f56eb47f00f32591a2e7c3bceea4
Nov 03 20:46:51.053261 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:51.053221    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9" event={"ID":"859533d1-0dc9-40dc-8baf-d494518ba039","Type":"ContainerStarted","Data":"179de4736fb202aa07a4f7bfcf73906738a7158f5850207ff1fc966c01019457"}
Nov 03 20:46:51.053464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:51.053279    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9" event={"ID":"859533d1-0dc9-40dc-8baf-d494518ba039","Type":"ContainerStarted","Data":"2a47934365aafd7cb60c14e53da60635ab36f56eb47f00f32591a2e7c3bceea4"}
Nov 03 20:46:51.054687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:51.054664    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt" event={"ID":"e454dd07-e27d-4498-ac05-53ccf1f341b6","Type":"ContainerDied","Data":"2166996feea1584cff3ec7a32bc769057c738a9f165a8861bcadb328b415089b"}
Nov 03 20:46:51.054794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:51.054706    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2166996feea1584cff3ec7a32bc769057c738a9f165a8861bcadb328b415089b"
Nov 03 20:46:51.054794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:51.054676    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"
Nov 03 20:46:51.160523 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:51.160442    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9" podStartSLOduration=2.160422639 podStartE2EDuration="2.160422639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:51.159587407 +0000 UTC m=+1051.163546130" watchObservedRunningTime="2026-11-03 20:46:51.160422639 +0000 UTC m=+1051.164381368"
Nov 03 20:46:51.852152 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:51.852117    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:52.064887 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:52.064850    5196 generic.go:356] "Generic (PLEG): container finished" podID="859533d1-0dc9-40dc-8baf-d494518ba039" containerID="179de4736fb202aa07a4f7bfcf73906738a7158f5850207ff1fc966c01019457" exitCode=0
Nov 03 20:46:52.065088 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:52.064936    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9" event={"ID":"859533d1-0dc9-40dc-8baf-d494518ba039","Type":"ContainerDied","Data":"179de4736fb202aa07a4f7bfcf73906738a7158f5850207ff1fc966c01019457"}
Nov 03 20:46:53.245370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.245339    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:53.356206 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.356125    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859533d1-0dc9-40dc-8baf-d494518ba039-kube-api-access-fwvqw\" (UniqueName: \"kubernetes.io/projected/859533d1-0dc9-40dc-8baf-d494518ba039-kube-api-access-fwvqw\") pod \"859533d1-0dc9-40dc-8baf-d494518ba039\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") "
Nov 03 20:46:53.356384 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.356312    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-xks-ca\") pod \"859533d1-0dc9-40dc-8baf-d494518ba039\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") "
Nov 03 20:46:53.356384 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.356346    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-openshift-ca\") pod \"859533d1-0dc9-40dc-8baf-d494518ba039\" (UID: \"859533d1-0dc9-40dc-8baf-d494518ba039\") "
Nov 03 20:46:53.356780 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.356754    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-openshift-ca" pod "859533d1-0dc9-40dc-8baf-d494518ba039" (UID: "859533d1-0dc9-40dc-8baf-d494518ba039"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:53.358563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.358535    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-xks-ca" pod "859533d1-0dc9-40dc-8baf-d494518ba039" (UID: "859533d1-0dc9-40dc-8baf-d494518ba039"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:53.358848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.358825    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859533d1-0dc9-40dc-8baf-d494518ba039-kube-api-access-fwvqw" pod "859533d1-0dc9-40dc-8baf-d494518ba039" (UID: "859533d1-0dc9-40dc-8baf-d494518ba039"). InnerVolumeSpecName "kube-api-access-fwvqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:53.457818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.457780    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwvqw\" (UniqueName: \"kubernetes.io/projected/859533d1-0dc9-40dc-8baf-d494518ba039-kube-api-access-fwvqw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:53.457818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.457812    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:53.457818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:53.457822    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859533d1-0dc9-40dc-8baf-d494518ba039-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:54.083221 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.083192    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"
Nov 03 20:46:54.083221 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.083212    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9" event={"ID":"859533d1-0dc9-40dc-8baf-d494518ba039","Type":"ContainerDied","Data":"2a47934365aafd7cb60c14e53da60635ab36f56eb47f00f32591a2e7c3bceea4"}
Nov 03 20:46:54.083479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.083271    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a47934365aafd7cb60c14e53da60635ab36f56eb47f00f32591a2e7c3bceea4"
Nov 03 20:46:54.412731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.412651    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"]
Nov 03 20:46:54.413334 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.413319    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e454dd07-e27d-4498-ac05-53ccf1f341b6" containerName="revoke-keys"
Nov 03 20:46:54.413412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.413336    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e454dd07-e27d-4498-ac05-53ccf1f341b6" containerName="revoke-keys"
Nov 03 20:46:54.413412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.413387    5196 cpu_manager.go:414] "containerMap: removing container" podUID="859533d1-0dc9-40dc-8baf-d494518ba039" containerName="revoke-keys"
Nov 03 20:46:54.413412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.413393    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859533d1-0dc9-40dc-8baf-d494518ba039" containerName="revoke-keys"
Nov 03 20:46:54.413510 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.413489    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e454dd07-e27d-4498-ac05-53ccf1f341b6" containerName="revoke-keys"
Nov 03 20:46:54.413510 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.413507    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="859533d1-0dc9-40dc-8baf-d494518ba039" containerName="revoke-keys"
Nov 03 20:46:54.417723 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.417698    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.467969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.467938    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:46:54.469386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.469349    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"]
Nov 03 20:46:54.469554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.469533    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wbt\" (UniqueName: \"kubernetes.io/projected/74351c3a-db06-4887-9edb-8edd08a43b37-kube-api-access-66wbt\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.469628 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.469612    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.469772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.469756    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.511718 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.511688    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:46:54.570777 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.570743    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.570926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.570841    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.570926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.570910    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wbt\" (UniqueName: \"kubernetes.io/projected/74351c3a-db06-4887-9edb-8edd08a43b37-kube-api-access-66wbt\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.571464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.571434    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.573411 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.573384    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.624403 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.624372    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wbt\" (UniqueName: \"kubernetes.io/projected/74351c3a-db06-4887-9edb-8edd08a43b37-kube-api-access-66wbt\") pod \"maas-api-revoke-sub-23e0d8c9-ls7d8\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:54.733497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:54.733418    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:55.116701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:55.116665    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"]
Nov 03 20:46:55.121520 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:46:55.121491    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74351c3a_db06_4887_9edb_8edd08a43b37.slice/crio-902a20c7a9e70f8d170cda97a5ad713ca77b4a76ea5c29b0616a11ef7767e5e6 WatchSource:0}: Error finding container 902a20c7a9e70f8d170cda97a5ad713ca77b4a76ea5c29b0616a11ef7767e5e6: Status 404 returned error can't find the container with id 902a20c7a9e70f8d170cda97a5ad713ca77b4a76ea5c29b0616a11ef7767e5e6
Nov 03 20:46:56.098980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:56.098945    5196 generic.go:356] "Generic (PLEG): container finished" podID="74351c3a-db06-4887-9edb-8edd08a43b37" containerID="0283ff85e3f6f3583e702cfdab48d757989dea255c73e45eae56fd5a90867c23" exitCode=0
Nov 03 20:46:56.099398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:56.099031    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8" event={"ID":"74351c3a-db06-4887-9edb-8edd08a43b37","Type":"ContainerDied","Data":"0283ff85e3f6f3583e702cfdab48d757989dea255c73e45eae56fd5a90867c23"}
Nov 03 20:46:56.099398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:56.099085    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8" event={"ID":"74351c3a-db06-4887-9edb-8edd08a43b37","Type":"ContainerStarted","Data":"902a20c7a9e70f8d170cda97a5ad713ca77b4a76ea5c29b0616a11ef7767e5e6"}
Nov 03 20:46:56.586430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:56.586396    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:57.266604 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.266575    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:57.403675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.403559    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-xks-ca\") pod \"74351c3a-db06-4887-9edb-8edd08a43b37\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") "
Nov 03 20:46:57.403871 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.403726    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74351c3a-db06-4887-9edb-8edd08a43b37-kube-api-access-66wbt\" (UniqueName: \"kubernetes.io/projected/74351c3a-db06-4887-9edb-8edd08a43b37-kube-api-access-66wbt\") pod \"74351c3a-db06-4887-9edb-8edd08a43b37\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") "
Nov 03 20:46:57.403871 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.403753    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-openshift-ca\") pod \"74351c3a-db06-4887-9edb-8edd08a43b37\" (UID: \"74351c3a-db06-4887-9edb-8edd08a43b37\") "
Nov 03 20:46:57.404335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.404304    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-openshift-ca" pod "74351c3a-db06-4887-9edb-8edd08a43b37" (UID: "74351c3a-db06-4887-9edb-8edd08a43b37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:57.406618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.406586    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-xks-ca" pod "74351c3a-db06-4887-9edb-8edd08a43b37" (UID: "74351c3a-db06-4887-9edb-8edd08a43b37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:57.406618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.406606    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74351c3a-db06-4887-9edb-8edd08a43b37-kube-api-access-66wbt" pod "74351c3a-db06-4887-9edb-8edd08a43b37" (UID: "74351c3a-db06-4887-9edb-8edd08a43b37"). InnerVolumeSpecName "kube-api-access-66wbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:57.505001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.504963    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wbt\" (UniqueName: \"kubernetes.io/projected/74351c3a-db06-4887-9edb-8edd08a43b37-kube-api-access-66wbt\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:57.505001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.504997    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:57.505243 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:57.505013    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74351c3a-db06-4887-9edb-8edd08a43b37-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:58.116025 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:58.115997    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"
Nov 03 20:46:58.116306 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:58.116037    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8" event={"ID":"74351c3a-db06-4887-9edb-8edd08a43b37","Type":"ContainerDied","Data":"902a20c7a9e70f8d170cda97a5ad713ca77b4a76ea5c29b0616a11ef7767e5e6"}
Nov 03 20:46:58.116306 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:46:58.116088    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="902a20c7a9e70f8d170cda97a5ad713ca77b4a76ea5c29b0616a11ef7767e5e6"
Nov 03 20:47:01.382201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:01.382168    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:04.860786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.860750    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"]
Nov 03 20:47:04.861455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.861440    5196 cpu_manager.go:414] "containerMap: removing container" podUID="74351c3a-db06-4887-9edb-8edd08a43b37" containerName="revoke-keys"
Nov 03 20:47:04.861507 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.861456    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74351c3a-db06-4887-9edb-8edd08a43b37" containerName="revoke-keys"
Nov 03 20:47:04.861620 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.861608    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="74351c3a-db06-4887-9edb-8edd08a43b37" containerName="revoke-keys"
Nov 03 20:47:04.864991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.864971    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:04.918462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.918420    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"]
Nov 03 20:47:04.959661 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.959627    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:04.959661 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.959663    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:47:04.995708 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.995674    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:04.995879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.995862    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:04.995955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:04.995897    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvq9c\" (UniqueName: \"kubernetes.io/projected/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-kube-api-access-tvq9c\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:05.097657 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.097601    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:05.097868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.097669    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvq9c\" (UniqueName: \"kubernetes.io/projected/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-kube-api-access-tvq9c\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:05.097868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.097773    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:05.098310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.098289    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:05.100450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.100424    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:05.152648 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.152552    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvq9c\" (UniqueName: \"kubernetes.io/projected/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-kube-api-access-tvq9c\") pod \"maas-api-revoke-sub-ae938b92-7469f\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:05.179273 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.179240    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:05.201641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.201602    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"]
Nov 03 20:47:05.206166 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.206132    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.268528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.268492    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"]
Nov 03 20:47:05.299773 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.299743    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdczs\" (UniqueName: \"kubernetes.io/projected/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-kube-api-access-pdczs\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.299937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.299797    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.300027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.300003    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.401659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.401620    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.401848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.401719    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdczs\" (UniqueName: \"kubernetes.io/projected/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-kube-api-access-pdczs\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.401848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.401787    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.402465 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.402443    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.404322 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.404273    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.455437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.455404    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdczs\" (UniqueName: \"kubernetes.io/projected/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-kube-api-access-pdczs\") pod \"maas-api-revoke-sub-ff1f420c-dpg8l\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.533062 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.533023    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:05.562052 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.561919    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"]
Nov 03 20:47:05.573290 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:05.573251    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb922f2e_feb3_4249_a4a4_052ae2f4c6a0.slice/crio-6d0eedeafcec049dc0969b22efd01f7422f79d14a78f7acd52bc2ec01671805e WatchSource:0}: Error finding container 6d0eedeafcec049dc0969b22efd01f7422f79d14a78f7acd52bc2ec01671805e: Status 404 returned error can't find the container with id 6d0eedeafcec049dc0969b22efd01f7422f79d14a78f7acd52bc2ec01671805e
Nov 03 20:47:05.894016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.893983    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"]
Nov 03 20:47:05.899528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.899507    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:05.917019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:05.916994    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"]
Nov 03 20:47:05.921294 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:05.921270    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37b9dfa_401a_476d_9387_50a7f6cd1d3f.slice/crio-1141bcbe9b118646f27e59b0d3a5952721f66443a5b7aa656f60e52e2a8be0e0 WatchSource:0}: Error finding container 1141bcbe9b118646f27e59b0d3a5952721f66443a5b7aa656f60e52e2a8be0e0: Status 404 returned error can't find the container with id 1141bcbe9b118646f27e59b0d3a5952721f66443a5b7aa656f60e52e2a8be0e0
Nov 03 20:47:06.008766 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.008719    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.008953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.008857    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.009025 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.008958    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.009025 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.009000    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.009197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.009033    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.009259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.009201    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.009259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.009244    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.009380 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.009362    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hsw\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-kube-api-access-b5hsw\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.009446 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.009431    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.091638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.091601    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"]
Nov 03 20:47:06.107923 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.107837    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:06.110308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110281    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110349    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110374    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110394    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110424    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110510    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110546    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110612    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hsw\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-kube-api-access-b5hsw\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110820 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110678    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110815    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.110875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.110832    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.111365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.111341    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.111625 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.111607    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.112026 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.111947    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.113304 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.113274    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.114075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.114046    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.164837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.164800    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.164990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.164923    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hsw\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-kube-api-access-b5hsw\") pod \"maas-default-gateway-openshift-default-696f47c6c-btmgg\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.187393 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.187359    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f" event={"ID":"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0","Type":"ContainerStarted","Data":"dbed79e3c2b96ef7f1d295ab371a097ef2930f21eaacf9c5d09652f2e31ae169"}
Nov 03 20:47:06.187570 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.187419    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f" event={"ID":"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0","Type":"ContainerStarted","Data":"6d0eedeafcec049dc0969b22efd01f7422f79d14a78f7acd52bc2ec01671805e"}
Nov 03 20:47:06.188732 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.188706    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l" event={"ID":"f37b9dfa-401a-476d-9387-50a7f6cd1d3f","Type":"ContainerStarted","Data":"1141bcbe9b118646f27e59b0d3a5952721f66443a5b7aa656f60e52e2a8be0e0"}
Nov 03 20:47:06.215464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.215432    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:06.295510 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.295448    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f" podStartSLOduration=2.295427623 podStartE2EDuration="2.295427623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:06.295040805 +0000 UTC m=+1066.298999546" watchObservedRunningTime="2026-11-03 20:47:06.295427623 +0000 UTC m=+1066.299386347"
Nov 03 20:47:06.595036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.595005    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"]
Nov 03 20:47:06.612209 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:06.612175    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463aab31_1e86_464e_97c4_d85d3a7d4f2a.slice/crio-9156cc8f3c2e0485a1e6d2a8419d0ad7fe3a0a618d751fe2dbd5818d67f736fd WatchSource:0}: Error finding container 9156cc8f3c2e0485a1e6d2a8419d0ad7fe3a0a618d751fe2dbd5818d67f736fd: Status 404 returned error can't find the container with id 9156cc8f3c2e0485a1e6d2a8419d0ad7fe3a0a618d751fe2dbd5818d67f736fd
Nov 03 20:47:06.615076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.615034    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:06.615184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.615152    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:06.615231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:06.615198    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:07.203408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.203306    5196 generic.go:356] "Generic (PLEG): container finished" podID="cb922f2e-feb3-4249-a4a4-052ae2f4c6a0" containerID="dbed79e3c2b96ef7f1d295ab371a097ef2930f21eaacf9c5d09652f2e31ae169" exitCode=0
Nov 03 20:47:07.203408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.203389    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f" event={"ID":"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0","Type":"ContainerDied","Data":"dbed79e3c2b96ef7f1d295ab371a097ef2930f21eaacf9c5d09652f2e31ae169"}
Nov 03 20:47:07.205954 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.205928    5196 generic.go:356] "Generic (PLEG): container finished" podID="f37b9dfa-401a-476d-9387-50a7f6cd1d3f" containerID="4595a15238223684d85ab8c2a86d5bc342d3459c74c5ea36c2e8c3e962358d02" exitCode=0
Nov 03 20:47:07.206126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.206017    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l" event={"ID":"f37b9dfa-401a-476d-9387-50a7f6cd1d3f","Type":"ContainerDied","Data":"4595a15238223684d85ab8c2a86d5bc342d3459c74c5ea36c2e8c3e962358d02"}
Nov 03 20:47:07.208005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.207980    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" event={"ID":"463aab31-1e86-464e-97c4-d85d3a7d4f2a","Type":"ContainerStarted","Data":"6742fc52080a7f35a474eaea9e7e4fceb139c500feba733865fd1516cdbef08f"}
Nov 03 20:47:07.208167 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.208025    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" event={"ID":"463aab31-1e86-464e-97c4-d85d3a7d4f2a","Type":"ContainerStarted","Data":"9156cc8f3c2e0485a1e6d2a8419d0ad7fe3a0a618d751fe2dbd5818d67f736fd"}
Nov 03 20:47:07.216524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.216501    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:07.218426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.218393    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:47:07.218572 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.218455    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 20:47:07.581499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:07.581433    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podStartSLOduration=2.581413462 podStartE2EDuration="2.581413462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:07.579943574 +0000 UTC m=+1067.583902322" watchObservedRunningTime="2026-11-03 20:47:07.581413462 +0000 UTC m=+1067.585372187"
Nov 03 20:47:08.216444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.215970    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:47:08.216444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.216045    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 20:47:08.429576 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.429551    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:08.476570 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.476501    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:08.478734 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.478714    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-kube-api-access-pdczs\" (UniqueName: \"kubernetes.io/projected/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-kube-api-access-pdczs\") pod \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") "
Nov 03 20:47:08.478838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.478754    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-xks-ca\") pod \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") "
Nov 03 20:47:08.478838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.478783    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-openshift-ca\") pod \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") "
Nov 03 20:47:08.478949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.478864    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-openshift-ca\") pod \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") "
Nov 03 20:47:08.478949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.478884    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-xks-ca\") pod \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\" (UID: \"f37b9dfa-401a-476d-9387-50a7f6cd1d3f\") "
Nov 03 20:47:08.479115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.478976    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-kube-api-access-tvq9c\" (UniqueName: \"kubernetes.io/projected/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-kube-api-access-tvq9c\") pod \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\" (UID: \"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0\") "
Nov 03 20:47:08.479282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.479249    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-openshift-ca" pod "f37b9dfa-401a-476d-9387-50a7f6cd1d3f" (UID: "f37b9dfa-401a-476d-9387-50a7f6cd1d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:08.479636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.479510    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-openshift-ca" pod "cb922f2e-feb3-4249-a4a4-052ae2f4c6a0" (UID: "cb922f2e-feb3-4249-a4a4-052ae2f4c6a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:08.479766 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.479734    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:08.481201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.481178    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-kube-api-access-pdczs" pod "f37b9dfa-401a-476d-9387-50a7f6cd1d3f" (UID: "f37b9dfa-401a-476d-9387-50a7f6cd1d3f"). InnerVolumeSpecName "kube-api-access-pdczs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:08.481287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.481273    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-kube-api-access-tvq9c" pod "cb922f2e-feb3-4249-a4a4-052ae2f4c6a0" (UID: "cb922f2e-feb3-4249-a4a4-052ae2f4c6a0"). InnerVolumeSpecName "kube-api-access-tvq9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:08.481714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.481685    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-xks-ca" pod "f37b9dfa-401a-476d-9387-50a7f6cd1d3f" (UID: "f37b9dfa-401a-476d-9387-50a7f6cd1d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:08.481800 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.481771    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-xks-ca" pod "cb922f2e-feb3-4249-a4a4-052ae2f4c6a0" (UID: "cb922f2e-feb3-4249-a4a4-052ae2f4c6a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:08.580936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.580885    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:08.580936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.580930    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:08.580936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.580945    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvq9c\" (UniqueName: \"kubernetes.io/projected/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-kube-api-access-tvq9c\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:08.581272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.580960    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdczs\" (UniqueName: \"kubernetes.io/projected/f37b9dfa-401a-476d-9387-50a7f6cd1d3f-kube-api-access-pdczs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:08.581272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:08.580974    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:09.216030 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:09.215992    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:47:09.216204 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:09.216056    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 20:47:09.224363 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:09.224339    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"
Nov 03 20:47:09.224784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:09.224374    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f" event={"ID":"cb922f2e-feb3-4249-a4a4-052ae2f4c6a0","Type":"ContainerDied","Data":"6d0eedeafcec049dc0969b22efd01f7422f79d14a78f7acd52bc2ec01671805e"}
Nov 03 20:47:09.224784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:09.224421    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d0eedeafcec049dc0969b22efd01f7422f79d14a78f7acd52bc2ec01671805e"
Nov 03 20:47:09.225914 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:09.225891    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"
Nov 03 20:47:09.226062 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:09.225997    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l" event={"ID":"f37b9dfa-401a-476d-9387-50a7f6cd1d3f","Type":"ContainerDied","Data":"1141bcbe9b118646f27e59b0d3a5952721f66443a5b7aa656f60e52e2a8be0e0"}
Nov 03 20:47:09.226062 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:09.226038    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1141bcbe9b118646f27e59b0d3a5952721f66443a5b7aa656f60e52e2a8be0e0"
Nov 03 20:47:10.209266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.209235    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"]
Nov 03 20:47:10.209887 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.209874    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f37b9dfa-401a-476d-9387-50a7f6cd1d3f" containerName="revoke-keys"
Nov 03 20:47:10.209946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.209889    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f37b9dfa-401a-476d-9387-50a7f6cd1d3f" containerName="revoke-keys"
Nov 03 20:47:10.209946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.209900    5196 cpu_manager.go:414] "containerMap: removing container" podUID="cb922f2e-feb3-4249-a4a4-052ae2f4c6a0" containerName="revoke-keys"
Nov 03 20:47:10.209946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.209906    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb922f2e-feb3-4249-a4a4-052ae2f4c6a0" containerName="revoke-keys"
Nov 03 20:47:10.210037 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.210016    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f37b9dfa-401a-476d-9387-50a7f6cd1d3f" containerName="revoke-keys"
Nov 03 20:47:10.210037 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.210025    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb922f2e-feb3-4249-a4a4-052ae2f4c6a0" containerName="revoke-keys"
Nov 03 20:47:10.213735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.213718    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.216568 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.216543    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:47:10.216690 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.216608    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 20:47:10.265474 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.264830    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:47:10.267205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.266272    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:10.290985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.281628    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"]
Nov 03 20:47:10.300448 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.300409    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.300617 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.300475    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvjp\" (UniqueName: \"kubernetes.io/projected/f49733fd-37e2-4a37-9066-f76314873be0-kube-api-access-9cvjp\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.300617 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.300563    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.401963 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.401924    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.402159 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.401982    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvjp\" (UniqueName: \"kubernetes.io/projected/f49733fd-37e2-4a37-9066-f76314873be0-kube-api-access-9cvjp\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.402159 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.402013    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.402701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.402683    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.404500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.404482    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.462309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.462228    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvjp\" (UniqueName: \"kubernetes.io/projected/f49733fd-37e2-4a37-9066-f76314873be0-kube-api-access-9cvjp\") pod \"maas-api-revoke-sub-84d53323-vqm9h\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.530186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.530149    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:10.961266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:10.961199    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"]
Nov 03 20:47:10.964296 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:10.964267    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49733fd_37e2_4a37_9066_f76314873be0.slice/crio-4406a830b6b631afda4a738fb6064b5d31e6616edc8bd5aa92a312e19bc17def WatchSource:0}: Error finding container 4406a830b6b631afda4a738fb6064b5d31e6616edc8bd5aa92a312e19bc17def: Status 404 returned error can't find the container with id 4406a830b6b631afda4a738fb6064b5d31e6616edc8bd5aa92a312e19bc17def
Nov 03 20:47:11.065759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:11.065727    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:11.216219 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:11.216126    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:47:11.216219 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:11.216191    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 20:47:11.271163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:11.271131    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h" event={"ID":"f49733fd-37e2-4a37-9066-f76314873be0","Type":"ContainerStarted","Data":"4406a830b6b631afda4a738fb6064b5d31e6616edc8bd5aa92a312e19bc17def"}
Nov 03 20:47:12.216182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:12.216149    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:47:12.216346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:12.216210    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 20:47:12.280332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:12.280294    5196 generic.go:356] "Generic (PLEG): container finished" podID="f49733fd-37e2-4a37-9066-f76314873be0" containerID="c0359997e2db51fb7ea51b91b5f5b023dc9af32faf4926680c6d6f935facbc23" exitCode=0
Nov 03 20:47:12.280714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:12.280343    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h" event={"ID":"f49733fd-37e2-4a37-9066-f76314873be0","Type":"ContainerDied","Data":"c0359997e2db51fb7ea51b91b5f5b023dc9af32faf4926680c6d6f935facbc23"}
Nov 03 20:47:13.216176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.216130    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:47:13.216386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.216204    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 20:47:13.460835 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.460808    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:13.536610 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.536519    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-openshift-ca\") pod \"f49733fd-37e2-4a37-9066-f76314873be0\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") "
Nov 03 20:47:13.536610 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.536584    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-xks-ca\") pod \"f49733fd-37e2-4a37-9066-f76314873be0\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") "
Nov 03 20:47:13.536796 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.536780    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49733fd-37e2-4a37-9066-f76314873be0-kube-api-access-9cvjp\" (UniqueName: \"kubernetes.io/projected/f49733fd-37e2-4a37-9066-f76314873be0-kube-api-access-9cvjp\") pod \"f49733fd-37e2-4a37-9066-f76314873be0\" (UID: \"f49733fd-37e2-4a37-9066-f76314873be0\") "
Nov 03 20:47:13.536972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.536948    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-openshift-ca" pod "f49733fd-37e2-4a37-9066-f76314873be0" (UID: "f49733fd-37e2-4a37-9066-f76314873be0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:13.537202 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.537182    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:13.538980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.538947    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-xks-ca" pod "f49733fd-37e2-4a37-9066-f76314873be0" (UID: "f49733fd-37e2-4a37-9066-f76314873be0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:13.539079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.539042    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49733fd-37e2-4a37-9066-f76314873be0-kube-api-access-9cvjp" pod "f49733fd-37e2-4a37-9066-f76314873be0" (UID: "f49733fd-37e2-4a37-9066-f76314873be0"). InnerVolumeSpecName "kube-api-access-9cvjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:13.638681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.638619    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvjp\" (UniqueName: \"kubernetes.io/projected/f49733fd-37e2-4a37-9066-f76314873be0-kube-api-access-9cvjp\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:13.638681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:13.638671    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49733fd-37e2-4a37-9066-f76314873be0-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:14.216390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.216349    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:47:14.216574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.216419    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 20:47:14.298446 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.298414    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"
Nov 03 20:47:14.298446 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.298436    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h" event={"ID":"f49733fd-37e2-4a37-9066-f76314873be0","Type":"ContainerDied","Data":"4406a830b6b631afda4a738fb6064b5d31e6616edc8bd5aa92a312e19bc17def"}
Nov 03 20:47:14.298646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.298484    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4406a830b6b631afda4a738fb6064b5d31e6616edc8bd5aa92a312e19bc17def"
Nov 03 20:47:14.410181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.410082    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"]
Nov 03 20:47:14.410842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.410821    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f49733fd-37e2-4a37-9066-f76314873be0" containerName="revoke-keys"
Nov 03 20:47:14.410953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.410844    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49733fd-37e2-4a37-9066-f76314873be0" containerName="revoke-keys"
Nov 03 20:47:14.411033 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.411019    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49733fd-37e2-4a37-9066-f76314873be0" containerName="revoke-keys"
Nov 03 20:47:14.415508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.415484    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.449357 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.449317    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.449569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.449544    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m872t\" (UniqueName: \"kubernetes.io/projected/88653563-4582-4c54-8175-e0f9d3a94bf7-kube-api-access-m872t\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.449750 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.449730    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.465246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.465199    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:14.465678 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.465316    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:47:14.466996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.466964    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"]
Nov 03 20:47:14.551659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.551614    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m872t\" (UniqueName: \"kubernetes.io/projected/88653563-4582-4c54-8175-e0f9d3a94bf7-kube-api-access-m872t\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.551843 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.551787    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.551902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.551839    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.552596 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.552571    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.554674 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.554643    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.606036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.606004    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m872t\" (UniqueName: \"kubernetes.io/projected/88653563-4582-4c54-8175-e0f9d3a94bf7-kube-api-access-m872t\") pod \"maas-api-revoke-sub-0fe9b41d-lhrdp\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:14.732506 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:14.732413    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:15.128946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:15.128910    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"]
Nov 03 20:47:15.150301 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:15.134435    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88653563_4582_4c54_8175_e0f9d3a94bf7.slice/crio-dbdec9c6627149c3425220d3e3474fed5c41ce084e264e8095e1ffcd0c9281ca WatchSource:0}: Error finding container dbdec9c6627149c3425220d3e3474fed5c41ce084e264e8095e1ffcd0c9281ca: Status 404 returned error can't find the container with id dbdec9c6627149c3425220d3e3474fed5c41ce084e264e8095e1ffcd0c9281ca
Nov 03 20:47:15.307716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:15.307679    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp" event={"ID":"88653563-4582-4c54-8175-e0f9d3a94bf7","Type":"ContainerStarted","Data":"dbdec9c6627149c3425220d3e3474fed5c41ce084e264e8095e1ffcd0c9281ca"}
Nov 03 20:47:15.364609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:15.364571    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:15.365070 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:15.365045    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:15.366190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:15.366166    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:47:15.882532 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:15.882490    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:16.316820 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:16.316786    5196 generic.go:356] "Generic (PLEG): container finished" podID="88653563-4582-4c54-8175-e0f9d3a94bf7" containerID="6378721dd83aaeebbbcfc4c11c625f1b6d8285392f2568b3f03c61a0f1b8e420" exitCode=0
Nov 03 20:47:16.317004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:16.316876    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp" event={"ID":"88653563-4582-4c54-8175-e0f9d3a94bf7","Type":"ContainerDied","Data":"6378721dd83aaeebbbcfc4c11c625f1b6d8285392f2568b3f03c61a0f1b8e420"}
Nov 03 20:47:17.495350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.495323    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:17.592349 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.592255    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-xks-ca\") pod \"88653563-4582-4c54-8175-e0f9d3a94bf7\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") "
Nov 03 20:47:17.592528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.592392    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-openshift-ca\") pod \"88653563-4582-4c54-8175-e0f9d3a94bf7\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") "
Nov 03 20:47:17.592595 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.592543    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88653563-4582-4c54-8175-e0f9d3a94bf7-kube-api-access-m872t\" (UniqueName: \"kubernetes.io/projected/88653563-4582-4c54-8175-e0f9d3a94bf7-kube-api-access-m872t\") pod \"88653563-4582-4c54-8175-e0f9d3a94bf7\" (UID: \"88653563-4582-4c54-8175-e0f9d3a94bf7\") "
Nov 03 20:47:17.592774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.592748    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-openshift-ca" pod "88653563-4582-4c54-8175-e0f9d3a94bf7" (UID: "88653563-4582-4c54-8175-e0f9d3a94bf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:17.592923 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.592907    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:17.594469 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.594446    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-xks-ca" pod "88653563-4582-4c54-8175-e0f9d3a94bf7" (UID: "88653563-4582-4c54-8175-e0f9d3a94bf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:17.594780 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.594758    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88653563-4582-4c54-8175-e0f9d3a94bf7-kube-api-access-m872t" pod "88653563-4582-4c54-8175-e0f9d3a94bf7" (UID: "88653563-4582-4c54-8175-e0f9d3a94bf7"). InnerVolumeSpecName "kube-api-access-m872t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:17.694547 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.694499    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m872t\" (UniqueName: \"kubernetes.io/projected/88653563-4582-4c54-8175-e0f9d3a94bf7-kube-api-access-m872t\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:17.694547 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:17.694533    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88653563-4582-4c54-8175-e0f9d3a94bf7-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:18.333002 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:18.332971    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"
Nov 03 20:47:18.333216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:18.333001    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp" event={"ID":"88653563-4582-4c54-8175-e0f9d3a94bf7","Type":"ContainerDied","Data":"dbdec9c6627149c3425220d3e3474fed5c41ce084e264e8095e1ffcd0c9281ca"}
Nov 03 20:47:18.333216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:18.333060    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbdec9c6627149c3425220d3e3474fed5c41ce084e264e8095e1ffcd0c9281ca"
Nov 03 20:47:18.486058 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:47:18.485967    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:47:20.004808 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.004774    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"]
Nov 03 20:47:20.005482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.005466    5196 cpu_manager.go:414] "containerMap: removing container" podUID="88653563-4582-4c54-8175-e0f9d3a94bf7" containerName="revoke-keys"
Nov 03 20:47:20.005534 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.005483    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88653563-4582-4c54-8175-e0f9d3a94bf7" containerName="revoke-keys"
Nov 03 20:47:20.005598 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.005589    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="88653563-4582-4c54-8175-e0f9d3a94bf7" containerName="revoke-keys"
Nov 03 20:47:20.009482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.009462    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.061747 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.061713    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"]
Nov 03 20:47:20.103251 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.103217    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:20.103422 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.103359    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:47:20.129754 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.129717    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.129951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.129770    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.129951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.129927    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrn6\" (UniqueName: \"kubernetes.io/projected/3d06932b-717a-4def-9cfb-a4e2372cd6e6-kube-api-access-vlrn6\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.231502 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.231467    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.231705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.231515    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.231705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.231577    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrn6\" (UniqueName: \"kubernetes.io/projected/3d06932b-717a-4def-9cfb-a4e2372cd6e6-kube-api-access-vlrn6\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.232148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.232075    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.234134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.234084    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.286043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.285960    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrn6\" (UniqueName: \"kubernetes.io/projected/3d06932b-717a-4def-9cfb-a4e2372cd6e6-kube-api-access-vlrn6\") pod \"maas-api-revoke-sub-149fac51-pv6ms\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.324461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.324417    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:20.581623 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.581573    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:20.705156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:20.705120    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"]
Nov 03 20:47:20.709721 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:20.709691    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d06932b_717a_4def_9cfb_a4e2372cd6e6.slice/crio-84d435b300fd94971af00f8666dfc2c6647df616852b1490e41876c2c2fa7a04 WatchSource:0}: Error finding container 84d435b300fd94971af00f8666dfc2c6647df616852b1490e41876c2c2fa7a04: Status 404 returned error can't find the container with id 84d435b300fd94971af00f8666dfc2c6647df616852b1490e41876c2c2fa7a04
Nov 03 20:47:21.358997 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.358962    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms" event={"ID":"3d06932b-717a-4def-9cfb-a4e2372cd6e6","Type":"ContainerStarted","Data":"fdb1fc65f4a2db68c4eedc188013f40a753ab7a799c989623f6eb4581228e134"}
Nov 03 20:47:21.359428 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.359027    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms" event={"ID":"3d06932b-717a-4def-9cfb-a4e2372cd6e6","Type":"ContainerStarted","Data":"84d435b300fd94971af00f8666dfc2c6647df616852b1490e41876c2c2fa7a04"}
Nov 03 20:47:21.470706 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.470638    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms" podStartSLOduration=2.470618707 podStartE2EDuration="2.470618707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:21.467535372 +0000 UTC m=+1081.471494119" watchObservedRunningTime="2026-11-03 20:47:21.470618707 +0000 UTC m=+1081.474577432"
Nov 03 20:47:21.520136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.520053    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"]
Nov 03 20:47:21.524855 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.524829    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.567361 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.567327    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"]
Nov 03 20:47:21.578620 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.578583    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:21.652224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.652138    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.652406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.652240    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s826\" (UniqueName: \"kubernetes.io/projected/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-kube-api-access-9s826\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.652406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.652316    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.753696 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.753656    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.753900 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.753735    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s826\" (UniqueName: \"kubernetes.io/projected/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-kube-api-access-9s826\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.753900 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.753778    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.754511 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.754482    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.756639 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.756614    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.809946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.809914    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s826\" (UniqueName: \"kubernetes.io/projected/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-kube-api-access-9s826\") pod \"maas-api-revoke-sub-896fc70e-r22bp\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:21.855203 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:21.843409    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:22.226252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:22.226163    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"]
Nov 03 20:47:22.230809 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:22.230778    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376dc5c5_3f5c_43f9_826c_aeb3298fd47a.slice/crio-2fb249fa38ad6b4f09db09ed6c460e825ba0d3c5f5114caebb60db721008ebac WatchSource:0}: Error finding container 2fb249fa38ad6b4f09db09ed6c460e825ba0d3c5f5114caebb60db721008ebac: Status 404 returned error can't find the container with id 2fb249fa38ad6b4f09db09ed6c460e825ba0d3c5f5114caebb60db721008ebac
Nov 03 20:47:22.369570 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:22.369531    5196 generic.go:356] "Generic (PLEG): container finished" podID="3d06932b-717a-4def-9cfb-a4e2372cd6e6" containerID="fdb1fc65f4a2db68c4eedc188013f40a753ab7a799c989623f6eb4581228e134" exitCode=0
Nov 03 20:47:22.370000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:22.369575    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms" event={"ID":"3d06932b-717a-4def-9cfb-a4e2372cd6e6","Type":"ContainerDied","Data":"fdb1fc65f4a2db68c4eedc188013f40a753ab7a799c989623f6eb4581228e134"}
Nov 03 20:47:22.371144 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:22.371123    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp" event={"ID":"376dc5c5-3f5c-43f9-826c-aeb3298fd47a","Type":"ContainerStarted","Data":"2fb249fa38ad6b4f09db09ed6c460e825ba0d3c5f5114caebb60db721008ebac"}
Nov 03 20:47:23.380935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.380900    5196 generic.go:356] "Generic (PLEG): container finished" podID="376dc5c5-3f5c-43f9-826c-aeb3298fd47a" containerID="8525a42d42a0ec4b7ba8723f46aad8ddac4860be3d979a28274717f9b18c1980" exitCode=0
Nov 03 20:47:23.381397 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.380991    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp" event={"ID":"376dc5c5-3f5c-43f9-826c-aeb3298fd47a","Type":"ContainerDied","Data":"8525a42d42a0ec4b7ba8723f46aad8ddac4860be3d979a28274717f9b18c1980"}
Nov 03 20:47:23.537443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.537413    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:23.676530 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.676499    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d06932b-717a-4def-9cfb-a4e2372cd6e6-kube-api-access-vlrn6\" (UniqueName: \"kubernetes.io/projected/3d06932b-717a-4def-9cfb-a4e2372cd6e6-kube-api-access-vlrn6\") pod \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") "
Nov 03 20:47:23.676758 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.676710    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-openshift-ca\") pod \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") "
Nov 03 20:47:23.676877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.676843    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-xks-ca\") pod \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\" (UID: \"3d06932b-717a-4def-9cfb-a4e2372cd6e6\") "
Nov 03 20:47:23.677245 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.677206    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-openshift-ca" pod "3d06932b-717a-4def-9cfb-a4e2372cd6e6" (UID: "3d06932b-717a-4def-9cfb-a4e2372cd6e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:23.677433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.677403    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:23.679049 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.679026    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d06932b-717a-4def-9cfb-a4e2372cd6e6-kube-api-access-vlrn6" pod "3d06932b-717a-4def-9cfb-a4e2372cd6e6" (UID: "3d06932b-717a-4def-9cfb-a4e2372cd6e6"). InnerVolumeSpecName "kube-api-access-vlrn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:23.679288 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.679272    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-xks-ca" pod "3d06932b-717a-4def-9cfb-a4e2372cd6e6" (UID: "3d06932b-717a-4def-9cfb-a4e2372cd6e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:23.778455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.778363    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrn6\" (UniqueName: \"kubernetes.io/projected/3d06932b-717a-4def-9cfb-a4e2372cd6e6-kube-api-access-vlrn6\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:23.778455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:23.778403    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d06932b-717a-4def-9cfb-a4e2372cd6e6-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:24.293045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.293009    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"]
Nov 03 20:47:24.293998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.293977    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3d06932b-717a-4def-9cfb-a4e2372cd6e6" containerName="revoke-keys"
Nov 03 20:47:24.294126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.294001    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d06932b-717a-4def-9cfb-a4e2372cd6e6" containerName="revoke-keys"
Nov 03 20:47:24.294126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.294121    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d06932b-717a-4def-9cfb-a4e2372cd6e6" containerName="revoke-keys"
Nov 03 20:47:24.297987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.297965    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.352623 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.352583    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"]
Nov 03 20:47:24.384725 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.384621    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzz5v\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-kube-api-access-kzz5v\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.384725 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.384696    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.385312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.384807    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.385312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.384857    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.385312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.384952    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.385312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.384983    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.385312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.385011    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.385312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.385160    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.385312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.385187    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.393763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.393733    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"
Nov 03 20:47:24.393763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.393750    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms" event={"ID":"3d06932b-717a-4def-9cfb-a4e2372cd6e6","Type":"ContainerDied","Data":"84d435b300fd94971af00f8666dfc2c6647df616852b1490e41876c2c2fa7a04"}
Nov 03 20:47:24.394000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.393803    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84d435b300fd94971af00f8666dfc2c6647df616852b1490e41876c2c2fa7a04"
Nov 03 20:47:24.394374 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.394331    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b119f2-openshift-default-dockercfg-d7fgm\""
Nov 03 20:47:24.486372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486336    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486379    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486406    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486446    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486463    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486482    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486523    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486556    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486696    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzz5v\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-kube-api-access-kzz5v\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.486922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.486900    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.487038 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.487016    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.487214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.487187    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.487327 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.487299    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.487485 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.487460    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.489477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.489456    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.489787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.489763    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.546946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.546915    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.547165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.547145    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzz5v\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-kube-api-access-kzz5v\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.575231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.575196    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:24.618989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.618950    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:24.688967 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.688931    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-xks-ca\") pod \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") "
Nov 03 20:47:24.689709 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.689034    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-openshift-ca\") pod \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") "
Nov 03 20:47:24.689709 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.689353    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-kube-api-access-9s826\" (UniqueName: \"kubernetes.io/projected/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-kube-api-access-9s826\") pod \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\" (UID: \"376dc5c5-3f5c-43f9-826c-aeb3298fd47a\") "
Nov 03 20:47:24.701250 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.696639    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-openshift-ca" pod "376dc5c5-3f5c-43f9-826c-aeb3298fd47a" (UID: "376dc5c5-3f5c-43f9-826c-aeb3298fd47a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:24.703042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.703005    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-kube-api-access-9s826" pod "376dc5c5-3f5c-43f9-826c-aeb3298fd47a" (UID: "376dc5c5-3f5c-43f9-826c-aeb3298fd47a"). InnerVolumeSpecName "kube-api-access-9s826". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:24.704827 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.704785    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-xks-ca" pod "376dc5c5-3f5c-43f9-826c-aeb3298fd47a" (UID: "376dc5c5-3f5c-43f9-826c-aeb3298fd47a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:24.795626 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.795591    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s826\" (UniqueName: \"kubernetes.io/projected/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-kube-api-access-9s826\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:24.795626 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.795620    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:24.795626 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:24.795631    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376dc5c5-3f5c-43f9-826c-aeb3298fd47a-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:25.052860 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.052783    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"]
Nov 03 20:47:25.056552 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:25.056519    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5e55a3_0cc1_48c1_a42e_700f06a2e0fe.slice/crio-9a77f3826b49283681773290832b2f6f296db04a1235aec8d42662dd9f4d5368 WatchSource:0}: Error finding container 9a77f3826b49283681773290832b2f6f296db04a1235aec8d42662dd9f4d5368: Status 404 returned error can't find the container with id 9a77f3826b49283681773290832b2f6f296db04a1235aec8d42662dd9f4d5368
Nov 03 20:47:25.059138 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.059063    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:25.059209 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.059189    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:25.059254 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.059222    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:25.380640 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.380602    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:25.403252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.403222    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"
Nov 03 20:47:25.403723 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.403226    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp" event={"ID":"376dc5c5-3f5c-43f9-826c-aeb3298fd47a","Type":"ContainerDied","Data":"2fb249fa38ad6b4f09db09ed6c460e825ba0d3c5f5114caebb60db721008ebac"}
Nov 03 20:47:25.403723 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.403338    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fb249fa38ad6b4f09db09ed6c460e825ba0d3c5f5114caebb60db721008ebac"
Nov 03 20:47:25.405546 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.405516    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" event={"ID":"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe","Type":"ContainerStarted","Data":"b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0"}
Nov 03 20:47:25.405676 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.405599    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" event={"ID":"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe","Type":"ContainerStarted","Data":"9a77f3826b49283681773290832b2f6f296db04a1235aec8d42662dd9f4d5368"}
Nov 03 20:47:25.517952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.517891    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podStartSLOduration=1.517876224 podStartE2EDuration="1.517876224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:25.514937243 +0000 UTC m=+1085.518895967" watchObservedRunningTime="2026-11-03 20:47:25.517876224 +0000 UTC m=+1085.521834949"
Nov 03 20:47:25.620027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:25.619990    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:26.177480 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.177446    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"]
Nov 03 20:47:26.178151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.178136    5196 cpu_manager.go:414] "containerMap: removing container" podUID="376dc5c5-3f5c-43f9-826c-aeb3298fd47a" containerName="revoke-keys"
Nov 03 20:47:26.178198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.178153    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376dc5c5-3f5c-43f9-826c-aeb3298fd47a" containerName="revoke-keys"
Nov 03 20:47:26.178286 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.178276    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="376dc5c5-3f5c-43f9-826c-aeb3298fd47a" containerName="revoke-keys"
Nov 03 20:47:26.182217 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.182195    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.232331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.232290    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:26.232514 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.232438    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:47:26.259223 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.253194    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"]
Nov 03 20:47:26.316006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.315959    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnj4g\" (UniqueName: \"kubernetes.io/projected/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-kube-api-access-mnj4g\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.316233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.316021    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.316233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.316144    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.417245 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.417211    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.417758 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.417252    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.417758 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.417390    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnj4g\" (UniqueName: \"kubernetes.io/projected/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-kube-api-access-mnj4g\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.417980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.417950    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.420079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.420058    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.472186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.472086    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnj4g\" (UniqueName: \"kubernetes.io/projected/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-kube-api-access-mnj4g\") pod \"maas-api-revoke-sub-3607f42c-gntbz\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.496869 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.496827    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:26.621155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.621075    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:47:26.621323 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.621180    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 20:47:26.902237 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:26.902190    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"]
Nov 03 20:47:26.906457 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:26.906414    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49799c6c_7fc4_4fd1_8f5d_861a0759d6b1.slice/crio-737cdcc2c8ddb3345040f127ea424036af273eb10344696a292fbab946b53b74 WatchSource:0}: Error finding container 737cdcc2c8ddb3345040f127ea424036af273eb10344696a292fbab946b53b74: Status 404 returned error can't find the container with id 737cdcc2c8ddb3345040f127ea424036af273eb10344696a292fbab946b53b74
Nov 03 20:47:27.422763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:27.422725    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz" event={"ID":"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1","Type":"ContainerStarted","Data":"737cdcc2c8ddb3345040f127ea424036af273eb10344696a292fbab946b53b74"}
Nov 03 20:47:27.619745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:27.619698    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:47:27.619898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:27.619757    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 20:47:28.432325 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:28.432295    5196 generic.go:356] "Generic (PLEG): container finished" podID="49799c6c-7fc4-4fd1-8f5d-861a0759d6b1" containerID="b401ee05f3d1899f4ac0052d61d2fab077cb37ba4baecf43872f087d828b6606" exitCode=0
Nov 03 20:47:28.432783 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:28.432386    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz" event={"ID":"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1","Type":"ContainerDied","Data":"b401ee05f3d1899f4ac0052d61d2fab077cb37ba4baecf43872f087d828b6606"}
Nov 03 20:47:28.619856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:28.619816    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:47:28.620042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:28.619876    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 20:47:29.620401 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.620366    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:47:29.620837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.620435    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 20:47:29.682898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.682858    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:29.783516 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.783428    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-kube-api-access-mnj4g\" (UniqueName: \"kubernetes.io/projected/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-kube-api-access-mnj4g\") pod \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") "
Nov 03 20:47:29.783698 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.783680    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-xks-ca\") pod \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") "
Nov 03 20:47:29.783778 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.783763    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-openshift-ca\") pod \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\" (UID: \"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1\") "
Nov 03 20:47:29.784144 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.784119    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-openshift-ca" pod "49799c6c-7fc4-4fd1-8f5d-861a0759d6b1" (UID: "49799c6c-7fc4-4fd1-8f5d-861a0759d6b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:29.785950 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.785922    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-kube-api-access-mnj4g" pod "49799c6c-7fc4-4fd1-8f5d-861a0759d6b1" (UID: "49799c6c-7fc4-4fd1-8f5d-861a0759d6b1"). InnerVolumeSpecName "kube-api-access-mnj4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:29.786055 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.785957    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-xks-ca" pod "49799c6c-7fc4-4fd1-8f5d-861a0759d6b1" (UID: "49799c6c-7fc4-4fd1-8f5d-861a0759d6b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:29.885302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.885263    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:29.885302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.885303    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:29.885525 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:29.885320    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnj4g\" (UniqueName: \"kubernetes.io/projected/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1-kube-api-access-mnj4g\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:30.163161 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.163110    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:47:30.296952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.296921    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:30.506259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.506154    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz" event={"ID":"49799c6c-7fc4-4fd1-8f5d-861a0759d6b1","Type":"ContainerDied","Data":"737cdcc2c8ddb3345040f127ea424036af273eb10344696a292fbab946b53b74"}
Nov 03 20:47:30.506586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.506567    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="737cdcc2c8ddb3345040f127ea424036af273eb10344696a292fbab946b53b74"
Nov 03 20:47:30.506898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.506883    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"
Nov 03 20:47:30.624959 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.619370    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:47:30.624959 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.619430    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 20:47:30.961127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.954631    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"]
Nov 03 20:47:30.962383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.962354    5196 cpu_manager.go:414] "containerMap: removing container" podUID="49799c6c-7fc4-4fd1-8f5d-861a0759d6b1" containerName="revoke-keys"
Nov 03 20:47:30.962536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.962521    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49799c6c-7fc4-4fd1-8f5d-861a0759d6b1" containerName="revoke-keys"
Nov 03 20:47:30.962838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.962820    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="49799c6c-7fc4-4fd1-8f5d-861a0759d6b1" containerName="revoke-keys"
Nov 03 20:47:30.994120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:30.973685    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.076465 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.076391    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"]
Nov 03 20:47:31.080153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.077451    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:47:31.080153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.078298    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:31.080153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.079496    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.080153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.079547    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.080153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.079621    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczfd\" (UniqueName: \"kubernetes.io/projected/176971a3-734f-4e74-8cb2-f37fdf336d43-kube-api-access-pczfd\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.190955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.187353    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.190955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.187443    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.190955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.187547    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczfd\" (UniqueName: \"kubernetes.io/projected/176971a3-734f-4e74-8cb2-f37fdf336d43-kube-api-access-pczfd\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.194472 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.194304    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.194472 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.194386    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.271922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.267874    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczfd\" (UniqueName: \"kubernetes.io/projected/176971a3-734f-4e74-8cb2-f37fdf336d43-kube-api-access-pczfd\") pod \"maas-api-revoke-sub-c5c08165-k77w2\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.472345 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.471255    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:31.621268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.620795    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:47:31.621268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.620875    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 20:47:31.886878 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:31.886837    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"]
Nov 03 20:47:31.888716 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:31.888679    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176971a3_734f_4e74_8cb2_f37fdf336d43.slice/crio-bc9cf240a01ed644c546bcc8d8a7668e867cf3ab7477527ecf27e34507763dd5 WatchSource:0}: Error finding container bc9cf240a01ed644c546bcc8d8a7668e867cf3ab7477527ecf27e34507763dd5: Status 404 returned error can't find the container with id bc9cf240a01ed644c546bcc8d8a7668e867cf3ab7477527ecf27e34507763dd5
Nov 03 20:47:32.290282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.290230    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"]
Nov 03 20:47:32.296732 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.296207    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.315582 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.315547    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.315721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.315607    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.315776 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.315734    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhkf\" (UniqueName: \"kubernetes.io/projected/aeae2e99-a84a-4046-a90c-8a576c38dc96-kube-api-access-8zhkf\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.353119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.350015    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"]
Nov 03 20:47:32.421119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.419848    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.421119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.419926    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhkf\" (UniqueName: \"kubernetes.io/projected/aeae2e99-a84a-4046-a90c-8a576c38dc96-kube-api-access-8zhkf\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.421119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.420183    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.421119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.421039    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.438116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.433249    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.481464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.481426    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhkf\" (UniqueName: \"kubernetes.io/projected/aeae2e99-a84a-4046-a90c-8a576c38dc96-kube-api-access-8zhkf\") pod \"maas-api-revoke-sub-b567725d-zx5dc\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.527774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.527678    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2" event={"ID":"176971a3-734f-4e74-8cb2-f37fdf336d43","Type":"ContainerStarted","Data":"bd55a82148a8e89d94548b6ee1cc95b3f5ca366333e1d5aa1b305351133e919c"}
Nov 03 20:47:32.527774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.527741    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2" event={"ID":"176971a3-734f-4e74-8cb2-f37fdf336d43","Type":"ContainerStarted","Data":"bc9cf240a01ed644c546bcc8d8a7668e867cf3ab7477527ecf27e34507763dd5"}
Nov 03 20:47:32.614039 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.614001    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:32.638340 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.638267    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2" podStartSLOduration=2.638241835 podStartE2EDuration="2.638241835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:32.635518792 +0000 UTC m=+1092.639477520" watchObservedRunningTime="2026-11-03 20:47:32.638241835 +0000 UTC m=+1092.642200560"
Nov 03 20:47:32.994250 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:32.994214    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"]
Nov 03 20:47:32.999921 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:32.999873    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeae2e99_a84a_4046_a90c_8a576c38dc96.slice/crio-cbc108672f134295d3660f10c8c429b71027a17bd21873a1cb4d0e747f6ecf2a WatchSource:0}: Error finding container cbc108672f134295d3660f10c8c429b71027a17bd21873a1cb4d0e747f6ecf2a: Status 404 returned error can't find the container with id cbc108672f134295d3660f10c8c429b71027a17bd21873a1cb4d0e747f6ecf2a
Nov 03 20:47:33.471696 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:33.471665    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:33.472713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:33.472683    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:33.474345 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:33.474323    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 20:47:33.537487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:33.537387    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc" event={"ID":"aeae2e99-a84a-4046-a90c-8a576c38dc96","Type":"ContainerStarted","Data":"9cd51a450a6afbe1262ce566241ecd88e1eafe4ab78188834764cb7349896824"}
Nov 03 20:47:33.537487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:33.537451    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc" event={"ID":"aeae2e99-a84a-4046-a90c-8a576c38dc96","Type":"ContainerStarted","Data":"cbc108672f134295d3660f10c8c429b71027a17bd21873a1cb4d0e747f6ecf2a"}
Nov 03 20:47:33.539226 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:33.539200    5196 generic.go:356] "Generic (PLEG): container finished" podID="176971a3-734f-4e74-8cb2-f37fdf336d43" containerID="bd55a82148a8e89d94548b6ee1cc95b3f5ca366333e1d5aa1b305351133e919c" exitCode=0
Nov 03 20:47:33.539369 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:33.539275    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2" event={"ID":"176971a3-734f-4e74-8cb2-f37fdf336d43","Type":"ContainerDied","Data":"bd55a82148a8e89d94548b6ee1cc95b3f5ca366333e1d5aa1b305351133e919c"}
Nov 03 20:47:33.808165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:33.808035    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc" podStartSLOduration=1.808015361 podStartE2EDuration="1.808015361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:33.804650697 +0000 UTC m=+1093.808609481" watchObservedRunningTime="2026-11-03 20:47:33.808015361 +0000 UTC m=+1093.811974085"
Nov 03 20:47:34.550174 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.550063    5196 generic.go:356] "Generic (PLEG): container finished" podID="aeae2e99-a84a-4046-a90c-8a576c38dc96" containerID="9cd51a450a6afbe1262ce566241ecd88e1eafe4ab78188834764cb7349896824" exitCode=0
Nov 03 20:47:34.550174 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.550148    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc" event={"ID":"aeae2e99-a84a-4046-a90c-8a576c38dc96","Type":"ContainerDied","Data":"9cd51a450a6afbe1262ce566241ecd88e1eafe4ab78188834764cb7349896824"}
Nov 03 20:47:34.717902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.717875    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:34.748209 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.748180    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-xks-ca\") pod \"176971a3-734f-4e74-8cb2-f37fdf336d43\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") "
Nov 03 20:47:34.748209 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.748218    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176971a3-734f-4e74-8cb2-f37fdf336d43-kube-api-access-pczfd\" (UniqueName: \"kubernetes.io/projected/176971a3-734f-4e74-8cb2-f37fdf336d43-kube-api-access-pczfd\") pod \"176971a3-734f-4e74-8cb2-f37fdf336d43\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") "
Nov 03 20:47:34.748421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.748307    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-openshift-ca\") pod \"176971a3-734f-4e74-8cb2-f37fdf336d43\" (UID: \"176971a3-734f-4e74-8cb2-f37fdf336d43\") "
Nov 03 20:47:34.748736 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.748704    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-openshift-ca" pod "176971a3-734f-4e74-8cb2-f37fdf336d43" (UID: "176971a3-734f-4e74-8cb2-f37fdf336d43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:34.750387 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.750361    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-xks-ca" pod "176971a3-734f-4e74-8cb2-f37fdf336d43" (UID: "176971a3-734f-4e74-8cb2-f37fdf336d43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:34.750645 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.750621    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176971a3-734f-4e74-8cb2-f37fdf336d43-kube-api-access-pczfd" pod "176971a3-734f-4e74-8cb2-f37fdf336d43" (UID: "176971a3-734f-4e74-8cb2-f37fdf336d43"). InnerVolumeSpecName "kube-api-access-pczfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:34.850071 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.850033    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pczfd\" (UniqueName: \"kubernetes.io/projected/176971a3-734f-4e74-8cb2-f37fdf336d43-kube-api-access-pczfd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:34.850335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.850311    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:34.850335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:34.850338    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176971a3-734f-4e74-8cb2-f37fdf336d43-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:35.180964 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.180870    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:35.559937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.559841    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"
Nov 03 20:47:35.559937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.559870    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2" event={"ID":"176971a3-734f-4e74-8cb2-f37fdf336d43","Type":"ContainerDied","Data":"bc9cf240a01ed644c546bcc8d8a7668e867cf3ab7477527ecf27e34507763dd5"}
Nov 03 20:47:35.559937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.559927    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc9cf240a01ed644c546bcc8d8a7668e867cf3ab7477527ecf27e34507763dd5"
Nov 03 20:47:35.727040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.727009    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:35.759639 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.759607    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeae2e99-a84a-4046-a90c-8a576c38dc96-kube-api-access-8zhkf\" (UniqueName: \"kubernetes.io/projected/aeae2e99-a84a-4046-a90c-8a576c38dc96-kube-api-access-8zhkf\") pod \"aeae2e99-a84a-4046-a90c-8a576c38dc96\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") "
Nov 03 20:47:35.759789 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.759674    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-xks-ca\") pod \"aeae2e99-a84a-4046-a90c-8a576c38dc96\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") "
Nov 03 20:47:35.759789 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.759703    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-openshift-ca\") pod \"aeae2e99-a84a-4046-a90c-8a576c38dc96\" (UID: \"aeae2e99-a84a-4046-a90c-8a576c38dc96\") "
Nov 03 20:47:35.760171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.760145    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-openshift-ca" pod "aeae2e99-a84a-4046-a90c-8a576c38dc96" (UID: "aeae2e99-a84a-4046-a90c-8a576c38dc96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:35.760277 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.760219    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:35.761922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.761896    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-xks-ca" pod "aeae2e99-a84a-4046-a90c-8a576c38dc96" (UID: "aeae2e99-a84a-4046-a90c-8a576c38dc96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:35.762005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.761990    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeae2e99-a84a-4046-a90c-8a576c38dc96-kube-api-access-8zhkf" pod "aeae2e99-a84a-4046-a90c-8a576c38dc96" (UID: "aeae2e99-a84a-4046-a90c-8a576c38dc96"). InnerVolumeSpecName "kube-api-access-8zhkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:35.861769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.861735    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeae2e99-a84a-4046-a90c-8a576c38dc96-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:35.861769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:35.861770    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zhkf\" (UniqueName: \"kubernetes.io/projected/aeae2e99-a84a-4046-a90c-8a576c38dc96-kube-api-access-8zhkf\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:36.569787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:36.569753    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"
Nov 03 20:47:36.570281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:36.569775    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc" event={"ID":"aeae2e99-a84a-4046-a90c-8a576c38dc96","Type":"ContainerDied","Data":"cbc108672f134295d3660f10c8c429b71027a17bd21873a1cb4d0e747f6ecf2a"}
Nov 03 20:47:36.570281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:36.569834    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbc108672f134295d3660f10c8c429b71027a17bd21873a1cb4d0e747f6ecf2a"
Nov 03 20:47:36.719354 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:36.719322    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:39.818049 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.818011    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"]
Nov 03 20:47:39.818788 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.818770    5196 cpu_manager.go:414] "containerMap: removing container" podUID="aeae2e99-a84a-4046-a90c-8a576c38dc96" containerName="revoke-keys"
Nov 03 20:47:39.818827 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.818789    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeae2e99-a84a-4046-a90c-8a576c38dc96" containerName="revoke-keys"
Nov 03 20:47:39.818827 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.818823    5196 cpu_manager.go:414] "containerMap: removing container" podUID="176971a3-734f-4e74-8cb2-f37fdf336d43" containerName="revoke-keys"
Nov 03 20:47:39.818914 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.818828    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176971a3-734f-4e74-8cb2-f37fdf336d43" containerName="revoke-keys"
Nov 03 20:47:39.818943 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.818935    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="176971a3-734f-4e74-8cb2-f37fdf336d43" containerName="revoke-keys"
Nov 03 20:47:39.818989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.818950    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeae2e99-a84a-4046-a90c-8a576c38dc96" containerName="revoke-keys"
Nov 03 20:47:39.823181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.823158    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:39.872586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.872553    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:39.872740 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.872674    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:47:39.875401 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.875374    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:39.929071 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.929018    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:39.929298 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.929239    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:39.929635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.929532    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qwn\" (UniqueName: \"kubernetes.io/projected/66a09464-8c6a-4d5d-802e-a7a87e38f904-kube-api-access-89qwn\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:39.934138 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:39.934079    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"]
Nov 03 20:47:40.031060 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.031011    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:40.031254 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.031174    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:40.031341 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.031317    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qwn\" (UniqueName: \"kubernetes.io/projected/66a09464-8c6a-4d5d-802e-a7a87e38f904-kube-api-access-89qwn\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:40.031879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.031858    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:40.033699 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.033679    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:40.104026 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.103944    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qwn\" (UniqueName: \"kubernetes.io/projected/66a09464-8c6a-4d5d-802e-a7a87e38f904-kube-api-access-89qwn\") pod \"maas-api-revoke-sub-449d4ab1-rjwzx\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:40.137267 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.137229    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:40.523085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.523007    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"]
Nov 03 20:47:40.527315 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:40.527286    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a09464_8c6a_4d5d_802e_a7a87e38f904.slice/crio-52fac686a9c8675e94f7a806c5428a422344c005f05833b023e9e01494fa05b3 WatchSource:0}: Error finding container 52fac686a9c8675e94f7a806c5428a422344c005f05833b023e9e01494fa05b3: Status 404 returned error can't find the container with id 52fac686a9c8675e94f7a806c5428a422344c005f05833b023e9e01494fa05b3
Nov 03 20:47:40.603585 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:40.603543    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx" event={"ID":"66a09464-8c6a-4d5d-802e-a7a87e38f904","Type":"ContainerStarted","Data":"52fac686a9c8675e94f7a806c5428a422344c005f05833b023e9e01494fa05b3"}
Nov 03 20:47:41.613949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:41.613911    5196 generic.go:356] "Generic (PLEG): container finished" podID="66a09464-8c6a-4d5d-802e-a7a87e38f904" containerID="b9eb3e483af41c1672fefabb7e96e37bcc369b895eb2ddd16aea5f9d47d35e0b" exitCode=0
Nov 03 20:47:41.614458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:41.614026    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx" event={"ID":"66a09464-8c6a-4d5d-802e-a7a87e38f904","Type":"ContainerDied","Data":"b9eb3e483af41c1672fefabb7e96e37bcc369b895eb2ddd16aea5f9d47d35e0b"}
Nov 03 20:47:41.794228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:41.794126    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:42.809239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.809209    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:42.867566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.867525    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a09464-8c6a-4d5d-802e-a7a87e38f904-kube-api-access-89qwn\" (UniqueName: \"kubernetes.io/projected/66a09464-8c6a-4d5d-802e-a7a87e38f904-kube-api-access-89qwn\") pod \"66a09464-8c6a-4d5d-802e-a7a87e38f904\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") "
Nov 03 20:47:42.867764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.867642    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-xks-ca\") pod \"66a09464-8c6a-4d5d-802e-a7a87e38f904\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") "
Nov 03 20:47:42.867764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.867686    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-openshift-ca\") pod \"66a09464-8c6a-4d5d-802e-a7a87e38f904\" (UID: \"66a09464-8c6a-4d5d-802e-a7a87e38f904\") "
Nov 03 20:47:42.868130 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.868068    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-openshift-ca" pod "66a09464-8c6a-4d5d-802e-a7a87e38f904" (UID: "66a09464-8c6a-4d5d-802e-a7a87e38f904"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:42.868271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.868250    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:42.869807 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.869782    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a09464-8c6a-4d5d-802e-a7a87e38f904-kube-api-access-89qwn" pod "66a09464-8c6a-4d5d-802e-a7a87e38f904" (UID: "66a09464-8c6a-4d5d-802e-a7a87e38f904"). InnerVolumeSpecName "kube-api-access-89qwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:42.869907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.869834    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-xks-ca" pod "66a09464-8c6a-4d5d-802e-a7a87e38f904" (UID: "66a09464-8c6a-4d5d-802e-a7a87e38f904"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:42.969403 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.969358    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a09464-8c6a-4d5d-802e-a7a87e38f904-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:42.969403 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:42.969402    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qwn\" (UniqueName: \"kubernetes.io/projected/66a09464-8c6a-4d5d-802e-a7a87e38f904-kube-api-access-89qwn\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:43.630765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:43.630722    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx" event={"ID":"66a09464-8c6a-4d5d-802e-a7a87e38f904","Type":"ContainerDied","Data":"52fac686a9c8675e94f7a806c5428a422344c005f05833b023e9e01494fa05b3"}
Nov 03 20:47:43.630950 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:43.630790    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52fac686a9c8675e94f7a806c5428a422344c005f05833b023e9e01494fa05b3"
Nov 03 20:47:43.630950 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:43.630734    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"
Nov 03 20:47:44.762816 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:44.762775    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:48.487704 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:47:48.487666    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:47:49.574257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:49.574219    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:54.358973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:54.358934    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:55.140807 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.140770    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"]
Nov 03 20:47:55.141459 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.141444    5196 cpu_manager.go:414] "containerMap: removing container" podUID="66a09464-8c6a-4d5d-802e-a7a87e38f904" containerName="revoke-keys"
Nov 03 20:47:55.141520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.141461    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a09464-8c6a-4d5d-802e-a7a87e38f904" containerName="revoke-keys"
Nov 03 20:47:55.141592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.141582    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a09464-8c6a-4d5d-802e-a7a87e38f904" containerName="revoke-keys"
Nov 03 20:47:55.146038 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.146015    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.197934 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.197899    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"]
Nov 03 20:47:55.239980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.239950    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:55.240197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.240049    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:47:55.322950 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.322917    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.323172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.322984    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtw59\" (UniqueName: \"kubernetes.io/projected/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-kube-api-access-qtw59\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.323172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.323163    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.335576 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.335539    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"]
Nov 03 20:47:55.339846 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.339827    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.389731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.389671    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a19980-openshift-default-dockercfg-nbrs7\""
Nov 03 20:47:55.395493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.395426    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"]
Nov 03 20:47:55.424295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.424257    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.424499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.424315    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtw59\" (UniqueName: \"kubernetes.io/projected/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-kube-api-access-qtw59\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.424499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.424384    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.424944 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.424924    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.427044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.427019    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.479798 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.479757    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtw59\" (UniqueName: \"kubernetes.io/projected/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-kube-api-access-qtw59\") pod \"maas-api-revoke-sub-e1eb5db4-jpslc\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.525982 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.525931    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.526150 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.526053    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.526150 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.526083    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.526287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.526179    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.526287 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.526243    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.526361 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.526347    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.526395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.526367    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c8e3ff-db29-4702-b133-7442d29d1ae8-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.526395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.526385    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkfmf\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-kube-api-access-vkfmf\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.526509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.526484    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627458    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627693 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627517    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627693 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627576    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627693 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627612    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627693 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627675    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627697    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627768    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627803    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c8e3ff-db29-4702-b133-7442d29d1ae8-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627827    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfmf\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-kube-api-access-vkfmf\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627866    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.627917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.627892    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.628250 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.628231    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.628364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.628341    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.628745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.628721    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c8e3ff-db29-4702-b133-7442d29d1ae8-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.630142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.630123    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.630274 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.630255    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.697933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.697858    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.698082 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.698059    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfmf\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-kube-api-access-vkfmf\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:55.761508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.761466    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:55.954558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:55.954471    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:56.140373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.140340    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"]
Nov 03 20:47:56.144966 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:56.144940    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c99c1a_549b_4580_b9e3_8f32fbb4d56b.slice/crio-29373344937e1574a6eafc185089efb83f08d0ba58c237f5d522a788a786d31f WatchSource:0}: Error finding container 29373344937e1574a6eafc185089efb83f08d0ba58c237f5d522a788a786d31f: Status 404 returned error can't find the container with id 29373344937e1574a6eafc185089efb83f08d0ba58c237f5d522a788a786d31f
Nov 03 20:47:56.336484 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.336451    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"]
Nov 03 20:47:56.352651 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:47:56.352620    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c8e3ff_db29_4702_b133_7442d29d1ae8.slice/crio-38426a367093d0985454e01b37f58e4ac74ba72decf45ea7341696288efe737c WatchSource:0}: Error finding container 38426a367093d0985454e01b37f58e4ac74ba72decf45ea7341696288efe737c: Status 404 returned error can't find the container with id 38426a367093d0985454e01b37f58e4ac74ba72decf45ea7341696288efe737c
Nov 03 20:47:56.355527 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.355486    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:56.355624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.355565    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:56.355624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.355608    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:47:56.738745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.738573    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc" event={"ID":"68c99c1a-549b-4580-b9e3-8f32fbb4d56b","Type":"ContainerStarted","Data":"641f7e54e93e4de492d3444721478ead8db085caed788ac8dfe8b2a93fe454e6"}
Nov 03 20:47:56.738745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.738644    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc" event={"ID":"68c99c1a-549b-4580-b9e3-8f32fbb4d56b","Type":"ContainerStarted","Data":"29373344937e1574a6eafc185089efb83f08d0ba58c237f5d522a788a786d31f"}
Nov 03 20:47:56.741144 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.741118    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" event={"ID":"89c8e3ff-db29-4702-b133-7442d29d1ae8","Type":"ContainerStarted","Data":"fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243"}
Nov 03 20:47:56.741296 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.741169    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" event={"ID":"89c8e3ff-db29-4702-b133-7442d29d1ae8","Type":"ContainerStarted","Data":"38426a367093d0985454e01b37f58e4ac74ba72decf45ea7341696288efe737c"}
Nov 03 20:47:56.847419 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.847367    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc" podStartSLOduration=1.847351489 podStartE2EDuration="1.847351489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:56.845582954 +0000 UTC m=+1116.849541696" watchObservedRunningTime="2026-11-03 20:47:56.847351489 +0000 UTC m=+1116.851310213"
Nov 03 20:47:56.962120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.955194    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:47:56.962650 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:56.962578    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podStartSLOduration=1.9625559209999999 podStartE2EDuration="1.962555921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:56.955736398 +0000 UTC m=+1116.959695155" watchObservedRunningTime="2026-11-03 20:47:56.962555921 +0000 UTC m=+1116.966514648"
Nov 03 20:47:57.769226 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:57.769189    5196 generic.go:356] "Generic (PLEG): container finished" podID="68c99c1a-549b-4580-b9e3-8f32fbb4d56b" containerID="641f7e54e93e4de492d3444721478ead8db085caed788ac8dfe8b2a93fe454e6" exitCode=0
Nov 03 20:47:57.769703 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:57.769271    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc" event={"ID":"68c99c1a-549b-4580-b9e3-8f32fbb4d56b","Type":"ContainerDied","Data":"641f7e54e93e4de492d3444721478ead8db085caed788ac8dfe8b2a93fe454e6"}
Nov 03 20:47:57.959335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:57.959299    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:47:57.959649 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:57.959611    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 20:47:58.965121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:58.964578    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:47:58.965121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:58.964642    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 20:47:59.006070 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.006042    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:59.077751 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.077716    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-xks-ca\") pod \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") "
Nov 03 20:47:59.077948 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.077781    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-openshift-ca\") pod \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") "
Nov 03 20:47:59.077948 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.077881    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-kube-api-access-qtw59\" (UniqueName: \"kubernetes.io/projected/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-kube-api-access-qtw59\") pod \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\" (UID: \"68c99c1a-549b-4580-b9e3-8f32fbb4d56b\") "
Nov 03 20:47:59.078134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.078115    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-openshift-ca" pod "68c99c1a-549b-4580-b9e3-8f32fbb4d56b" (UID: "68c99c1a-549b-4580-b9e3-8f32fbb4d56b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:59.078332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.078318    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:59.080016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.079993    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-xks-ca" pod "68c99c1a-549b-4580-b9e3-8f32fbb4d56b" (UID: "68c99c1a-549b-4580-b9e3-8f32fbb4d56b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:59.080246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.080223    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-kube-api-access-qtw59" pod "68c99c1a-549b-4580-b9e3-8f32fbb4d56b" (UID: "68c99c1a-549b-4580-b9e3-8f32fbb4d56b"). InnerVolumeSpecName "kube-api-access-qtw59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:59.180264 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.180218    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:59.180264 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.180255    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtw59\" (UniqueName: \"kubernetes.io/projected/68c99c1a-549b-4580-b9e3-8f32fbb4d56b-kube-api-access-qtw59\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:59.209971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.209928    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:59.797478 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.797449    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"
Nov 03 20:47:59.797478 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.797452    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc" event={"ID":"68c99c1a-549b-4580-b9e3-8f32fbb4d56b","Type":"ContainerDied","Data":"29373344937e1574a6eafc185089efb83f08d0ba58c237f5d522a788a786d31f"}
Nov 03 20:47:59.797704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.797501    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29373344937e1574a6eafc185089efb83f08d0ba58c237f5d522a788a786d31f"
Nov 03 20:47:59.959575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.959532    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:47:59.959787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:47:59.959598    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 20:48:00.957564 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:00.957523    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:48:00.958225 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:00.958191    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 20:48:01.958589 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:01.958545    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:48:01.959289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:01.959246    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 20:48:02.955528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:02.955494    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:48:02.955691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:02.955553    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 20:48:03.959697 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:03.959664    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:48:03.960228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:03.960208    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:48:03.960879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:03.960863    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 20:48:04.090542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:04.090506    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:08.587568 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:08.587529    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:48:08.783504 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:08.783470    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:09.546065 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.546027    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"]
Nov 03 20:48:09.546730 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.546716    5196 cpu_manager.go:414] "containerMap: removing container" podUID="68c99c1a-549b-4580-b9e3-8f32fbb4d56b" containerName="revoke-keys"
Nov 03 20:48:09.546771 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.546732    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c99c1a-549b-4580-b9e3-8f32fbb4d56b" containerName="revoke-keys"
Nov 03 20:48:09.546872 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.546864    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c99c1a-549b-4580-b9e3-8f32fbb4d56b" containerName="revoke-keys"
Nov 03 20:48:09.551933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.551914    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611244 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611206    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611257    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611323    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611435    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611605    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611635    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxk7\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-kube-api-access-vrxk7\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611662    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611944 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611761    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.611944 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.611825    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.648421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.648383    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-984cbc-openshift-default-dockercfg-qwh2c\""
Nov 03 20:48:09.655816 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.655779    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"]
Nov 03 20:48:09.712742 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.712697    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.712922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.712832    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.712922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.712858    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxk7\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-kube-api-access-vrxk7\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.712922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.712885    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713061 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.712944    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713061 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713048    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713106    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713129    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713179    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713202    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713208    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713484 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713407    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713553 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713526    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.713693 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.713666    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.715517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.715493    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.715928 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.715908    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.766839 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.766810    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.766974 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.766938    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxk7\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-kube-api-access-vrxk7\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:09.867360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:09.867322    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:10.302012 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:10.301934    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"]
Nov 03 20:48:10.305558 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:48:10.305528    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9f05b7_5f08_4c4f_ba48_4465dd06eef1.slice/crio-8647d5a823c3b2f8259df1c0f48db278a7ed539b177a61a9daa635e786e26d1a WatchSource:0}: Error finding container 8647d5a823c3b2f8259df1c0f48db278a7ed539b177a61a9daa635e786e26d1a: Status 404 returned error can't find the container with id 8647d5a823c3b2f8259df1c0f48db278a7ed539b177a61a9daa635e786e26d1a
Nov 03 20:48:10.308330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:10.308290    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:48:10.308528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:10.308497    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:48:10.308629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:10.308547    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:48:10.390888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:10.390849    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:10.899170 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:10.899128    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" event={"ID":"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1","Type":"ContainerStarted","Data":"a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33"}
Nov 03 20:48:10.899582 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:10.899190    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" event={"ID":"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1","Type":"ContainerStarted","Data":"8647d5a823c3b2f8259df1c0f48db278a7ed539b177a61a9daa635e786e26d1a"}
Nov 03 20:48:10.902240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:10.902212    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:11.015621 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:11.015566    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" podStartSLOduration=2.015551087 podStartE2EDuration="2.015551087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:11.015268264 +0000 UTC m=+1131.019227009" watchObservedRunningTime="2026-11-03 20:48:11.015551087 +0000 UTC m=+1131.019509810"
Nov 03 20:48:11.098572 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:11.098537    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:11.868441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:11.868394    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:11.870491 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:11.870459    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:48:11.870642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:11.870512    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 20:48:12.867915 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:12.867881    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:48:12.868448 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:12.868399    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 20:48:13.614767 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:13.614728    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:13.868689 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:13.868592    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:48:13.868689 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:13.868667    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 20:48:14.868437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:14.868396    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:48:14.868664 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:14.868466    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 20:48:15.867828 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:15.867791    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:48:15.868338 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:15.868199    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 20:48:17.108768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:17.108610    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:17.109343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:17.109316    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:17.110675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:17.110643    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 20:48:18.471429 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:18.471394    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:18.486923 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:48:18.486882    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:48:23.236627 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:23.236588    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:23.429234 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:23.429193    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:28.076305 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:28.076267    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:28.334802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:28.334710    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:32.509859 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:32.509824    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:48:32.794437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:32.794322    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:36.871669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:36.871621    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:37.565403 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:37.565359    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:42.148669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:42.148611    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:44.008804 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:44.008764    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:45.138666 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.138638    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"]
Nov 03 20:48:45.143774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.143744    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.201926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.201891    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"]
Nov 03 20:48:45.239449 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.239413    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"distinct-w1-41aa21-kserve-self-signed-certs\""
Nov 03 20:48:45.285498 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.285447    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:45.308888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.308852    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tls-certs\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.309075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.308903    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vrt\" (UniqueName: \"kubernetes.io/projected/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-kube-api-access-l6vrt\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.309167 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.309071    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-home\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.309240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.309212    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-model-cache\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.309336 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.309320    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tmp-dir\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.309420 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.309399    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-dshm\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.410557 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.410448    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tls-certs\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.410557 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.410503    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vrt\" (UniqueName: \"kubernetes.io/projected/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-kube-api-access-l6vrt\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.410801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.410580    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-home\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.410801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.410635    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-model-cache\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.410801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.410718    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tmp-dir\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.410801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.410767    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-dshm\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.411141 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.411115    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-home\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.411232 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.411138    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-model-cache\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.411291 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.411222    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tmp-dir\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.413251 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.413225    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-dshm\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.413462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.413445    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tls-certs\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.466692 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.466654    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vrt\" (UniqueName: \"kubernetes.io/projected/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-kube-api-access-l6vrt\") pod \"distinct-w1-41aa21-kserve-76d4997fcb-7crd7\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:45.757520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:45.757432    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:46.207214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:46.207171    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"]
Nov 03 20:48:46.216590 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:46.216554    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" event={"ID":"0fe9d04c-600c-4be8-82b4-a2a80b23f05d","Type":"ContainerStarted","Data":"4ba5d7d862bcaa19720da9e2bd7a92d7647a980efcfb2d3002536ce157ae57d6"}
Nov 03 20:48:47.187075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:47.187040    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:47.228163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:47.228116    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" event={"ID":"0fe9d04c-600c-4be8-82b4-a2a80b23f05d","Type":"ContainerStarted","Data":"bc4377b760a348c2c5870b4b4ff25c100554a68ad782a586cd82947b16b15774"}
Nov 03 20:48:47.340458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:47.340399    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podStartSLOduration=2.340379355 podStartE2EDuration="2.340379355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:47.335380071 +0000 UTC m=+1167.339338808" watchObservedRunningTime="2026-11-03 20:48:47.340379355 +0000 UTC m=+1167.344338079"
Nov 03 20:48:47.758140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:47.758072    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:48.486115 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:48:48.486058    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:48:51.984651 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:51.984608    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:55.757680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:55.757587    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:55.770672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:55.770647    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:56.317554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:56.317517    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:48:56.783309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:56.783253    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:58.608737 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:48:58.608698    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:49:00.627934 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:00.627897    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:01.576370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:01.576340    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:06.381047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:06.381001    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:11.288856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:11.288816    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:16.065224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:16.065188    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:17.433528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:17.433495    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:18.484926 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:49:18.484887    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:49:20.284456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:20.284428    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:49:20.290972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:20.290950    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:49:20.295385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:20.295366    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:49:20.301317 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:20.301297    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:49:20.781690 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:20.781653    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:25.565217 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:25.565168    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:27.697038 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:27.696996    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:49:30.369789 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:30.369752    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:33.883340 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:33.883290    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:35.052462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:35.052432    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:39.802176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:39.802138    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:42.542625 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:42.542587    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:44.586153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:44.586075    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:48.485723 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:49:48.485682    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:49:49.281436 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:49.281395    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:53.983835 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:53.983791    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:56.758558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.758521    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"]
Nov 03 20:49:56.763184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.763162    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:56.815700 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.815665    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"]
Nov 03 20:49:56.856970 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.856940    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1ec892\"/\"distinct-2-w5-1ec892-kserve-self-signed-certs\""
Nov 03 20:49:56.859584 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.859548    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:56.935116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.935064    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtgb\" (UniqueName: \"kubernetes.io/projected/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-kube-api-access-rxtgb\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:56.935299 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.935214    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tmp-dir\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:56.935299 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.935274    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-home\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:56.935377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.935299    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-model-cache\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:56.935377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.935327    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tls-certs\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:56.935444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:56.935379    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-dshm\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.036627 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.036535    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-dshm\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.036810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.036670    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtgb\" (UniqueName: \"kubernetes.io/projected/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-kube-api-access-rxtgb\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.036810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.036706    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tmp-dir\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.036810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.036724    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-home\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.036810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.036741    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-model-cache\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.036810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.036763    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tls-certs\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.037171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.037145    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tmp-dir\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.037248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.037197    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-model-cache\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.037248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.037204    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-home\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.039087 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.039064    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-dshm\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.039442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.039421    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tls-certs\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.090888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.090859    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtgb\" (UniqueName: \"kubernetes.io/projected/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-kube-api-access-rxtgb\") pod \"distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") " pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.376083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.376046    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:49:57.756536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.756295    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"]
Nov 03 20:49:57.759779 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:49:57.759743    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cdca4ca_b9ea_4576_bc2b_21231f397ec9.slice/crio-be03cdb78253a1a8594557d914e2e5fe8412fd2c99ff8b1ad68dee860d8dbd90 WatchSource:0}: Error finding container be03cdb78253a1a8594557d914e2e5fe8412fd2c99ff8b1ad68dee860d8dbd90: Status 404 returned error can't find the container with id be03cdb78253a1a8594557d914e2e5fe8412fd2c99ff8b1ad68dee860d8dbd90
Nov 03 20:49:57.818560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:57.818531    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" event={"ID":"9cdca4ca-b9ea-4576-bc2b-21231f397ec9","Type":"ContainerStarted","Data":"be03cdb78253a1a8594557d914e2e5fe8412fd2c99ff8b1ad68dee860d8dbd90"}
Nov 03 20:49:58.755668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:58.755628    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:58.829416 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:58.829380    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" event={"ID":"9cdca4ca-b9ea-4576-bc2b-21231f397ec9","Type":"ContainerStarted","Data":"9932ca54c21cb6ad197995a3d7c50b3353cad1c675024dd57bcf16f5379ac5e1"}
Nov 03 20:49:58.944566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:58.944497    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podStartSLOduration=2.9444726599999997 podStartE2EDuration="2.94447266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:58.938519684 +0000 UTC m=+1238.942478411" watchObservedRunningTime="2026-11-03 20:49:58.94447266 +0000 UTC m=+1238.948431400"
Nov 03 20:49:59.376534 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:49:59.376502    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:50:03.448044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:03.447999    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:03.643139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:03.643073    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:07.376859 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:07.376814    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:50:07.390057 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:07.390020    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:50:07.932585 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:07.932554    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:50:08.255619 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:08.255528    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:08.949623 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:08.949590    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:50:09.455138 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.455047    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"]
Nov 03 20:50:09.461254 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.461222    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.493658 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.493622    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8fv\" (UniqueName: \"kubernetes.io/projected/5b32de33-fd09-4b7d-9753-021355e3f33d-kube-api-access-zt8fv\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.493867 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.493763    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.493867 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.493846    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.507362 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.507323    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"]
Nov 03 20:50:09.546372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.546329    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:50:09.546544 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.546382    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:50:09.594980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.594943    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8fv\" (UniqueName: \"kubernetes.io/projected/5b32de33-fd09-4b7d-9753-021355e3f33d-kube-api-access-zt8fv\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.595228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.595035    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.595228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.595078    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.595774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.595744    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.597829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.597795    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.650046 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.650013    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8fv\" (UniqueName: \"kubernetes.io/projected/5b32de33-fd09-4b7d-9753-021355e3f33d-kube-api-access-zt8fv\") pod \"maas-api-revoke-sub-b08f1e87-4jmmn\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:09.784358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:09.784275    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:10.167077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:10.167042    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"]
Nov 03 20:50:10.169326 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:50:10.169290    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b32de33_fd09_4b7d_9753_021355e3f33d.slice/crio-86fc2d01a8af4f688acd41422d6e0d7ad8143b3a1950d2af75467df00c8d0e9f WatchSource:0}: Error finding container 86fc2d01a8af4f688acd41422d6e0d7ad8143b3a1950d2af75467df00c8d0e9f: Status 404 returned error can't find the container with id 86fc2d01a8af4f688acd41422d6e0d7ad8143b3a1950d2af75467df00c8d0e9f
Nov 03 20:50:10.942798 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:10.942763    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn" event={"ID":"5b32de33-fd09-4b7d-9753-021355e3f33d","Type":"ContainerStarted","Data":"d3df53ecd04946e7245a24771161705a90fa544e3f08760fa48777d5715a58b5"}
Nov 03 20:50:10.943020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:10.942818    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn" event={"ID":"5b32de33-fd09-4b7d-9753-021355e3f33d","Type":"ContainerStarted","Data":"86fc2d01a8af4f688acd41422d6e0d7ad8143b3a1950d2af75467df00c8d0e9f"}
Nov 03 20:50:11.051488 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:11.051428    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn" podStartSLOduration=2.051413952 podStartE2EDuration="2.051413952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:11.049378628 +0000 UTC m=+1251.053337370" watchObservedRunningTime="2026-11-03 20:50:11.051413952 +0000 UTC m=+1251.055372675"
Nov 03 20:50:11.952780 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:11.952745    5196 generic.go:356] "Generic (PLEG): container finished" podID="5b32de33-fd09-4b7d-9753-021355e3f33d" containerID="d3df53ecd04946e7245a24771161705a90fa544e3f08760fa48777d5715a58b5" exitCode=0
Nov 03 20:50:11.953222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:11.952825    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn" event={"ID":"5b32de33-fd09-4b7d-9753-021355e3f33d","Type":"ContainerDied","Data":"d3df53ecd04946e7245a24771161705a90fa544e3f08760fa48777d5715a58b5"}
Nov 03 20:50:13.062644 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.062578    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:13.137722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.137693    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:13.251432 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.251337    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-openshift-ca\") pod \"5b32de33-fd09-4b7d-9753-021355e3f33d\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") "
Nov 03 20:50:13.251629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.251548    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b32de33-fd09-4b7d-9753-021355e3f33d-kube-api-access-zt8fv\" (UniqueName: \"kubernetes.io/projected/5b32de33-fd09-4b7d-9753-021355e3f33d-kube-api-access-zt8fv\") pod \"5b32de33-fd09-4b7d-9753-021355e3f33d\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") "
Nov 03 20:50:13.251629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.251583    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-xks-ca\") pod \"5b32de33-fd09-4b7d-9753-021355e3f33d\" (UID: \"5b32de33-fd09-4b7d-9753-021355e3f33d\") "
Nov 03 20:50:13.251812 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.251790    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-openshift-ca" pod "5b32de33-fd09-4b7d-9753-021355e3f33d" (UID: "5b32de33-fd09-4b7d-9753-021355e3f33d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:13.252134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.252077    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:13.254015 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.253973    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-xks-ca" pod "5b32de33-fd09-4b7d-9753-021355e3f33d" (UID: "5b32de33-fd09-4b7d-9753-021355e3f33d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:50:13.254147 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.253985    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b32de33-fd09-4b7d-9753-021355e3f33d-kube-api-access-zt8fv" pod "5b32de33-fd09-4b7d-9753-021355e3f33d" (UID: "5b32de33-fd09-4b7d-9753-021355e3f33d"). InnerVolumeSpecName "kube-api-access-zt8fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:13.352948 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.352907    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt8fv\" (UniqueName: \"kubernetes.io/projected/5b32de33-fd09-4b7d-9753-021355e3f33d-kube-api-access-zt8fv\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:13.352948 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.352936    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b32de33-fd09-4b7d-9753-021355e3f33d-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:13.969786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.969748    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn" event={"ID":"5b32de33-fd09-4b7d-9753-021355e3f33d","Type":"ContainerDied","Data":"86fc2d01a8af4f688acd41422d6e0d7ad8143b3a1950d2af75467df00c8d0e9f"}
Nov 03 20:50:13.969786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.969771    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"
Nov 03 20:50:13.970006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:13.969833    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86fc2d01a8af4f688acd41422d6e0d7ad8143b3a1950d2af75467df00c8d0e9f"
Nov 03 20:50:17.762214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:17.762173    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:18.485202 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:50:18.485167    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:50:20.197696 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.197663    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"]
Nov 03 20:50:20.198416 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.198396    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5b32de33-fd09-4b7d-9753-021355e3f33d" containerName="revoke-keys"
Nov 03 20:50:20.198416 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.198416    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b32de33-fd09-4b7d-9753-021355e3f33d" containerName="revoke-keys"
Nov 03 20:50:20.198560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.198551    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b32de33-fd09-4b7d-9753-021355e3f33d" containerName="revoke-keys"
Nov 03 20:50:20.202359 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.202340    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.253755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.253723    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"]
Nov 03 20:50:20.295086 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.295052    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:50:20.295263 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.295176    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:50:20.341930 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.341888    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xc2\" (UniqueName: \"kubernetes.io/projected/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-kube-api-access-x2xc2\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.342190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.342168    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.342266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.342212    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.443327 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.443290    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.443327 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.443330    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.443560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.443387    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xc2\" (UniqueName: \"kubernetes.io/projected/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-kube-api-access-x2xc2\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.444233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.444079    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.445856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.445837    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.497623 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.497546    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xc2\" (UniqueName: \"kubernetes.io/projected/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-kube-api-access-x2xc2\") pod \"maas-api-revoke-sub-3e5929db-nlzgd\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.516405 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.516373    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:20.896936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:20.896907    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"]
Nov 03 20:50:20.899887 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:50:20.899854    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b37974d_46c9_42a9_96bc_1e8bd9f8f83a.slice/crio-59d93627f795c8fd190764e39d217e58c9a011ea10261db9dbc2bd7729613076 WatchSource:0}: Error finding container 59d93627f795c8fd190764e39d217e58c9a011ea10261db9dbc2bd7729613076: Status 404 returned error can't find the container with id 59d93627f795c8fd190764e39d217e58c9a011ea10261db9dbc2bd7729613076
Nov 03 20:50:21.027103 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:21.027053    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd" event={"ID":"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a","Type":"ContainerStarted","Data":"59d93627f795c8fd190764e39d217e58c9a011ea10261db9dbc2bd7729613076"}
Nov 03 20:50:22.036590 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:22.036551    5196 generic.go:356] "Generic (PLEG): container finished" podID="3b37974d-46c9-42a9-96bc-1e8bd9f8f83a" containerID="cb97502cf8813a11eafe2312af3f77e63f171b350196c0c610432fda83402ed3" exitCode=0
Nov 03 20:50:22.036985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:22.036639    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd" event={"ID":"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a","Type":"ContainerDied","Data":"cb97502cf8813a11eafe2312af3f77e63f171b350196c0c610432fda83402ed3"}
Nov 03 20:50:22.562521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:22.562475    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:23.233589 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.233563    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:23.377306 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.377267    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-kube-api-access-x2xc2\" (UniqueName: \"kubernetes.io/projected/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-kube-api-access-x2xc2\") pod \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") "
Nov 03 20:50:23.377517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.377432    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-openshift-ca\") pod \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") "
Nov 03 20:50:23.377517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.377463    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-xks-ca\") pod \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\" (UID: \"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a\") "
Nov 03 20:50:23.377893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.377862    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-openshift-ca" pod "3b37974d-46c9-42a9-96bc-1e8bd9f8f83a" (UID: "3b37974d-46c9-42a9-96bc-1e8bd9f8f83a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:23.379769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.379737    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-xks-ca" pod "3b37974d-46c9-42a9-96bc-1e8bd9f8f83a" (UID: "3b37974d-46c9-42a9-96bc-1e8bd9f8f83a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:50:23.379902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.379767    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-kube-api-access-x2xc2" pod "3b37974d-46c9-42a9-96bc-1e8bd9f8f83a" (UID: "3b37974d-46c9-42a9-96bc-1e8bd9f8f83a"). InnerVolumeSpecName "kube-api-access-x2xc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:23.479442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.479395    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xc2\" (UniqueName: \"kubernetes.io/projected/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-kube-api-access-x2xc2\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:23.479442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.479436    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:23.479687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:23.479475    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:24.054026 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.053942    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"
Nov 03 20:50:24.054321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.053969    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd" event={"ID":"3b37974d-46c9-42a9-96bc-1e8bd9f8f83a","Type":"ContainerDied","Data":"59d93627f795c8fd190764e39d217e58c9a011ea10261db9dbc2bd7729613076"}
Nov 03 20:50:24.054410 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.054341    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59d93627f795c8fd190764e39d217e58c9a011ea10261db9dbc2bd7729613076"
Nov 03 20:50:24.566127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.566065    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"]
Nov 03 20:50:24.567233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.567208    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3b37974d-46c9-42a9-96bc-1e8bd9f8f83a" containerName="revoke-keys"
Nov 03 20:50:24.567233 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.567234    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b37974d-46c9-42a9-96bc-1e8bd9f8f83a" containerName="revoke-keys"
Nov 03 20:50:24.567504 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.567488    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b37974d-46c9-42a9-96bc-1e8bd9f8f83a" containerName="revoke-keys"
Nov 03 20:50:24.572188 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.572166    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.621060 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.621027    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"]
Nov 03 20:50:24.662909 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.662870    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1ec892-openshift-default-dockercfg-dds29\""
Nov 03 20:50:24.698461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698412    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.698648 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698478    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbq5\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-kube-api-access-zpbq5\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.698648 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698608    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.698761 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698650    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.698824 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698766    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a8017f-edde-4061-836f-43dc01941446-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.698880 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698833    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a8017f-edde-4061-836f-43dc01941446-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.698880 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698872    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.698960 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698917    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.698960 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.698943    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.799856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.799817    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.799856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.799863    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.799907    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a8017f-edde-4061-836f-43dc01941446-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.799943    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a8017f-edde-4061-836f-43dc01941446-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.799978    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800006    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800041    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800178    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800232    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbq5\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-kube-api-access-zpbq5\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800384    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800505    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800528    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800743 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800725    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.800828 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.800806    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a8017f-edde-4061-836f-43dc01941446-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.802442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.802406    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.802777 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.802756    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a8017f-edde-4061-836f-43dc01941446-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.853905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.853875    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.854114 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.854009    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbq5\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-kube-api-access-zpbq5\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:24.887607 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:24.887567    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:25.321859 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:25.321833    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"]
Nov 03 20:50:25.325156 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:50:25.325127    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a8017f_edde_4061_836f_43dc01941446.slice/crio-1d4ff53037a42c0e5027e33cef7c9565f33f7604b3b851f85838ae8755ddfbf3 WatchSource:0}: Error finding container 1d4ff53037a42c0e5027e33cef7c9565f33f7604b3b851f85838ae8755ddfbf3: Status 404 returned error can't find the container with id 1d4ff53037a42c0e5027e33cef7c9565f33f7604b3b851f85838ae8755ddfbf3
Nov 03 20:50:25.327876 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:25.327842    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:50:25.327963 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:25.327914    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:50:25.327963 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:25.327941    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:50:26.072890 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:26.072854    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" event={"ID":"e1a8017f-edde-4061-836f-43dc01941446","Type":"ContainerStarted","Data":"1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61"}
Nov 03 20:50:26.073364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:26.072913    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" event={"ID":"e1a8017f-edde-4061-836f-43dc01941446","Type":"ContainerStarted","Data":"1d4ff53037a42c0e5027e33cef7c9565f33f7604b3b851f85838ae8755ddfbf3"}
Nov 03 20:50:26.186343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:26.186284    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podStartSLOduration=2.186270426 podStartE2EDuration="2.186270426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:26.182775297 +0000 UTC m=+1266.186734039" watchObservedRunningTime="2026-11-03 20:50:26.186270426 +0000 UTC m=+1266.190229200"
Nov 03 20:50:26.310162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:26.310128    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:26.888629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:26.888595    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:26.890595 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:26.890565    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 20:50:26.890744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:26.890620    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 20:50:27.360856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:27.360821    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:27.888834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:27.888797    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 20:50:27.889017 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:27.888856    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 20:50:28.888151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:28.888081    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 20:50:28.888576 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:28.888174    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 20:50:29.888905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:29.888865    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 20:50:29.889373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:29.888924    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 20:50:30.888528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:30.888489    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 20:50:30.888796 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:30.888565    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 20:50:31.888245 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:31.888205    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 20:50:31.888656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:31.888287    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 20:50:32.178985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:32.178898    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:32.892840 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:32.892809    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:32.893335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:32.893116    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:32.893889 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:32.893866    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 20:50:33.059361 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:33.059323    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"]
Nov 03 20:50:33.065375 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:33.065329    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84daddc3-bc59v"]
Nov 03 20:50:33.253822 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:33.253708    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:34.049142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:34.049105    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"]
Nov 03 20:50:34.053490 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:34.053464    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-983f4439-5nxwf"]
Nov 03 20:50:34.110381 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:34.110347    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35581491-a0a2-4e82-82bb-d3469692ee77" path="/var/lib/kubelet/pods/35581491-a0a2-4e82-82bb-d3469692ee77/volumes"
Nov 03 20:50:34.110882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:34.110867    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99883928-fe8a-4d39-806a-b9da6db79547" path="/var/lib/kubelet/pods/99883928-fe8a-4d39-806a-b9da6db79547/volumes"
Nov 03 20:50:36.964246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:36.964204    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:38.272534 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.272494    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"]
Nov 03 20:50:38.277674 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.277647    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.328606 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.328574    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"]
Nov 03 20:50:38.369647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.369612    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:50:38.369647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.369621    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:50:38.390815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.390782    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9jlt\" (UniqueName: \"kubernetes.io/projected/86db3616-4842-433e-adc9-61d51fe488da-kube-api-access-j9jlt\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.391028 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.390846    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3616-4842-433e-adc9-61d51fe488da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.391167 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.391081    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3616-4842-433e-adc9-61d51fe488da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.492713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.492674    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3616-4842-433e-adc9-61d51fe488da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.492897 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.492754    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9jlt\" (UniqueName: \"kubernetes.io/projected/86db3616-4842-433e-adc9-61d51fe488da-kube-api-access-j9jlt\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.492897 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.492798    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3616-4842-433e-adc9-61d51fe488da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.493633 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.493564    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3616-4842-433e-adc9-61d51fe488da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.495395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.495372    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3616-4842-433e-adc9-61d51fe488da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.546919 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.546839    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9jlt\" (UniqueName: \"kubernetes.io/projected/86db3616-4842-433e-adc9-61d51fe488da-kube-api-access-j9jlt\") pod \"maas-api-revoke-sub-496d699c-gvcp4\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.593767 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.593732    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:38.982293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:38.982252    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"]
Nov 03 20:50:38.986488 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:50:38.986454    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86db3616_4842_433e_adc9_61d51fe488da.slice/crio-f538d750ed9184a15c853f1d956342b481a2692a818b162d7adb56f4c1282c40 WatchSource:0}: Error finding container f538d750ed9184a15c853f1d956342b481a2692a818b162d7adb56f4c1282c40: Status 404 returned error can't find the container with id f538d750ed9184a15c853f1d956342b481a2692a818b162d7adb56f4c1282c40
Nov 03 20:50:39.182879 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:39.182844    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4" event={"ID":"86db3616-4842-433e-adc9-61d51fe488da","Type":"ContainerStarted","Data":"f538d750ed9184a15c853f1d956342b481a2692a818b162d7adb56f4c1282c40"}
Nov 03 20:50:40.193184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.193057    5196 generic.go:356] "Generic (PLEG): container finished" podID="86db3616-4842-433e-adc9-61d51fe488da" containerID="5cea8e4b394908f00e93fc11355269cfe9f5131541cefc87a9c3b196b9184a71" exitCode=0
Nov 03 20:50:40.193184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.193139    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4" event={"ID":"86db3616-4842-433e-adc9-61d51fe488da","Type":"ContainerDied","Data":"5cea8e4b394908f00e93fc11355269cfe9f5131541cefc87a9c3b196b9184a71"}
Nov 03 20:50:40.740278 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.740240    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:50:40.855219 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.855184    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"]
Nov 03 20:50:40.859805 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.859783    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:40.914678 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.914641    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b119f2\"/\"distinct-2-w2-b119f2-kserve-self-signed-certs\""
Nov 03 20:50:40.916008 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.915981    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"]
Nov 03 20:50:40.927834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.927776    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-dshm\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:40.928043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.927846    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eee8222-1165-47be-ab0f-237afafa9d07-tls-certs\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:40.928043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.927888    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-tmp-dir\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:40.928043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.927951    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-model-cache\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:40.928043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.928036    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-home\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:40.928329 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.928207    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbrr\" (UniqueName: \"kubernetes.io/projected/3eee8222-1165-47be-ab0f-237afafa9d07-kube-api-access-mzbrr\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:40.953748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:40.953715    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:41.029707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.029617    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbrr\" (UniqueName: \"kubernetes.io/projected/3eee8222-1165-47be-ab0f-237afafa9d07-kube-api-access-mzbrr\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.029885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.029720    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-dshm\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.029885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.029761    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eee8222-1165-47be-ab0f-237afafa9d07-tls-certs\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.029885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.029796    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-tmp-dir\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.029885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.029856    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-model-cache\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.030134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.029945    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-home\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.030402 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.030370    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-tmp-dir\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.030499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.030418    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-home\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.030675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.030648    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-model-cache\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.032556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.032526    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-dshm\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.033469 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.033447    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eee8222-1165-47be-ab0f-237afafa9d07-tls-certs\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.084242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.084196    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbrr\" (UniqueName: \"kubernetes.io/projected/3eee8222-1165-47be-ab0f-237afafa9d07-kube-api-access-mzbrr\") pod \"distinct-2-w2-b119f2-kserve-f9f788895-jzd2n\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") " pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.175845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.175803    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:41.494198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.494165    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:41.539431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.539380    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86db3616-4842-433e-adc9-61d51fe488da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3616-4842-433e-adc9-61d51fe488da-maas-api-xks-ca\") pod \"86db3616-4842-433e-adc9-61d51fe488da\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") "
Nov 03 20:50:41.539639 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.539537    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86db3616-4842-433e-adc9-61d51fe488da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3616-4842-433e-adc9-61d51fe488da-maas-api-openshift-ca\") pod \"86db3616-4842-433e-adc9-61d51fe488da\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") "
Nov 03 20:50:41.539705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.539651    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86db3616-4842-433e-adc9-61d51fe488da-kube-api-access-j9jlt\" (UniqueName: \"kubernetes.io/projected/86db3616-4842-433e-adc9-61d51fe488da-kube-api-access-j9jlt\") pod \"86db3616-4842-433e-adc9-61d51fe488da\" (UID: \"86db3616-4842-433e-adc9-61d51fe488da\") "
Nov 03 20:50:41.540508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.540372    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86db3616-4842-433e-adc9-61d51fe488da-maas-api-openshift-ca" pod "86db3616-4842-433e-adc9-61d51fe488da" (UID: "86db3616-4842-433e-adc9-61d51fe488da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:41.541861 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.541828    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"]
Nov 03 20:50:41.542776 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.542753    5196 cpu_manager.go:414] "containerMap: removing container" podUID="86db3616-4842-433e-adc9-61d51fe488da" containerName="revoke-keys"
Nov 03 20:50:41.542776 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.542775    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db3616-4842-433e-adc9-61d51fe488da" containerName="revoke-keys"
Nov 03 20:50:41.542987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.542967    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db3616-4842-433e-adc9-61d51fe488da" containerName="revoke-keys"
Nov 03 20:50:41.545937 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.545886    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86db3616-4842-433e-adc9-61d51fe488da-kube-api-access-j9jlt" pod "86db3616-4842-433e-adc9-61d51fe488da" (UID: "86db3616-4842-433e-adc9-61d51fe488da"). InnerVolumeSpecName "kube-api-access-j9jlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:41.550835 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.550809    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.555773 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.555705    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86db3616-4842-433e-adc9-61d51fe488da-maas-api-xks-ca" pod "86db3616-4842-433e-adc9-61d51fe488da" (UID: "86db3616-4842-433e-adc9-61d51fe488da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:50:41.641024 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.640985    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641024 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641028    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70311607-242b-4964-ab22-0a73105e0ac0-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641048    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641131    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641177    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgb4x\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-kube-api-access-rgb4x\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641212    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641238    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70311607-242b-4964-ab22-0a73105e0ac0-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641269    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641316    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.641741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641370    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9jlt\" (UniqueName: \"kubernetes.io/projected/86db3616-4842-433e-adc9-61d51fe488da-kube-api-access-j9jlt\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:41.641741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641397    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3616-4842-433e-adc9-61d51fe488da-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:41.641741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.641413    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3616-4842-433e-adc9-61d51fe488da-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:41.643714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.643681    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"]
Nov 03 20:50:41.666991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.666952    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"]
Nov 03 20:50:41.670753 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:50:41.670716    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eee8222_1165_47be_ab0f_237afafa9d07.slice/crio-a9bb3c1e5bc8a522ea8a36762532e9bfa542be4d8947fbd9208d2a791e1f5075 WatchSource:0}: Error finding container a9bb3c1e5bc8a522ea8a36762532e9bfa542be4d8947fbd9208d2a791e1f5075: Status 404 returned error can't find the container with id a9bb3c1e5bc8a522ea8a36762532e9bfa542be4d8947fbd9208d2a791e1f5075
Nov 03 20:50:41.727155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.724245    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:41.743925 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.743895    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.743933    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70311607-242b-4964-ab22-0a73105e0ac0-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.743971    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744015    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744137    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744169    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70311607-242b-4964-ab22-0a73105e0ac0-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744198    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744326    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744407    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgb4x\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-kube-api-access-rgb4x\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744498 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744474    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744496    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744640 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744614    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744700 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744681    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.744844 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.744823    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70311607-242b-4964-ab22-0a73105e0ac0-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.747166 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.747138    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.747283 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.747215    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70311607-242b-4964-ab22-0a73105e0ac0-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.798352 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.798142    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.798510 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.798392    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgb4x\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-kube-api-access-rgb4x\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:41.928077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:41.927990    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:42.215925 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.215824    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" event={"ID":"3eee8222-1165-47be-ab0f-237afafa9d07","Type":"ContainerStarted","Data":"e5be51f5a587da6dbb3f8e9956458c96017512a2569d66a4b74c45950f34e915"}
Nov 03 20:50:42.215925 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.215875    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" event={"ID":"3eee8222-1165-47be-ab0f-237afafa9d07","Type":"ContainerStarted","Data":"a9bb3c1e5bc8a522ea8a36762532e9bfa542be4d8947fbd9208d2a791e1f5075"}
Nov 03 20:50:42.217363 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.217336    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"
Nov 03 20:50:42.217479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.217384    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4" event={"ID":"86db3616-4842-433e-adc9-61d51fe488da","Type":"ContainerDied","Data":"f538d750ed9184a15c853f1d956342b481a2692a818b162d7adb56f4c1282c40"}
Nov 03 20:50:42.217479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.217428    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f538d750ed9184a15c853f1d956342b481a2692a818b162d7adb56f4c1282c40"
Nov 03 20:50:42.325534 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.325481    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podStartSLOduration=2.325466651 podStartE2EDuration="2.325466651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:42.322766331 +0000 UTC m=+1282.326725090" watchObservedRunningTime="2026-11-03 20:50:42.325466651 +0000 UTC m=+1282.329425374"
Nov 03 20:50:42.360083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.360048    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"]
Nov 03 20:50:42.364959 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:50:42.364931    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70311607_242b_4964_ab22_0a73105e0ac0.slice/crio-a06b4f0378c986610571f013e3af4031938d22574a79fa5598674d7db8163114 WatchSource:0}: Error finding container a06b4f0378c986610571f013e3af4031938d22574a79fa5598674d7db8163114: Status 404 returned error can't find the container with id a06b4f0378c986610571f013e3af4031938d22574a79fa5598674d7db8163114
Nov 03 20:50:42.367659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.367617    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:50:42.367762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.367689    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:50:42.367762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.367725    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:50:42.792467 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:42.791559    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:43.176384 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:43.176348    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:43.227428 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:43.227382    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" event={"ID":"70311607-242b-4964-ab22-0a73105e0ac0","Type":"ContainerStarted","Data":"30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20"}
Nov 03 20:50:43.227642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:43.227441    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" event={"ID":"70311607-242b-4964-ab22-0a73105e0ac0","Type":"ContainerStarted","Data":"a06b4f0378c986610571f013e3af4031938d22574a79fa5598674d7db8163114"}
Nov 03 20:50:43.364174 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:43.364107    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podStartSLOduration=2.364070706 podStartE2EDuration="2.364070706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:43.359479114 +0000 UTC m=+1283.363437838" watchObservedRunningTime="2026-11-03 20:50:43.364070706 +0000 UTC m=+1283.368029432"
Nov 03 20:50:43.928966 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:43.928934    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:43.930577 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:43.930547    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 20:50:43.930701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:43.930609    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 20:50:44.928526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:44.928493    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 20:50:44.928696 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:44.928557    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 20:50:45.928701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:45.928657    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 20:50:45.929226 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:45.928729    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 20:50:46.505322 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:46.505289    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:46.858135 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:46.857311    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:46.929253 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:46.929203    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 20:50:46.929753 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:46.929276    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 20:50:47.928397 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:47.928362    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 20:50:47.928605 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:47.928427    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 20:50:48.488372 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:50:48.488328    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:50:48.928463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:48.928428    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 20:50:48.928683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:48.928500    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 20:50:49.047042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:49.047008    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"]
Nov 03 20:50:49.086481 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:49.086444    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffb86cd-c6n8q"]
Nov 03 20:50:49.933892 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:49.933860    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:49.934379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:49.934356    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:49.935139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:49.935118    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 20:50:50.112102 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:50.112055    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c616e8e7-02ee-4fe8-89b2-dc1bd2e23482" path="/var/lib/kubelet/pods/c616e8e7-02ee-4fe8-89b2-dc1bd2e23482/volumes"
Nov 03 20:50:51.176604 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:51.176568    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:51.189745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:51.189717    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:51.332378 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:51.332348    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:50:51.358264 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:51.358220    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:53.814415 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:53.814382    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"]
Nov 03 20:50:53.820438 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:53.820408    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:53.870501 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:53.870392    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:50:53.870501 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:53.870471    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:50:53.873367 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:53.873328    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"]
Nov 03 20:50:53.907480 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:53.907431    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:53.907480 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:53.907472    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6z2\" (UniqueName: \"kubernetes.io/projected/6c684363-f63c-41d5-af34-d3232e1feef0-kube-api-access-bp6z2\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:53.907718 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:53.907651    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:54.009149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.009087    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:54.009310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.009218    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:54.009310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.009240    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6z2\" (UniqueName: \"kubernetes.io/projected/6c684363-f63c-41d5-af34-d3232e1feef0-kube-api-access-bp6z2\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:54.009907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.009877    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:54.011859 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.011832    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:54.063276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.063239    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6z2\" (UniqueName: \"kubernetes.io/projected/6c684363-f63c-41d5-af34-d3232e1feef0-kube-api-access-bp6z2\") pod \"maas-api-revoke-sub-f58eda9f-qz6zg\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:54.140833 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.140785    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:54.142614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.142587    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"]
Nov 03 20:50:54.147603 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.147582    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.190415 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.190382    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"]
Nov 03 20:50:54.197757 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.197721    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-41aa21\"/\"distinct-2-w1-41aa21-kserve-self-signed-certs\""
Nov 03 20:50:54.212709 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.212669    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-home\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.212886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.212777    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tls-certs\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.212886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.212848    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tmp-dir\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.213019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.212899    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtpp\" (UniqueName: \"kubernetes.io/projected/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-kube-api-access-dgtpp\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.213159 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.213139    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-dshm\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.213320 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.213302    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-model-cache\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.239272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.239239    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.314443    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tls-certs\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.314515    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tmp-dir\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.314548    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtpp\" (UniqueName: \"kubernetes.io/projected/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-kube-api-access-dgtpp\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.314725    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-dshm\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.314841    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-model-cache\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.314897    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-home\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.315510    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-home\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.316520    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-model-cache\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.317495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.317078    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tmp-dir\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.332456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.332416    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-dshm\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.333859 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.333748    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tls-certs\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.371522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.371488    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtpp\" (UniqueName: \"kubernetes.io/projected/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-kube-api-access-dgtpp\") pod \"distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") " pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.500711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.500584    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:54.538301 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.538255    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"]
Nov 03 20:50:54.906787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:54.906761    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"]
Nov 03 20:50:54.910032 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:50:54.909991    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod784d401d_fb0d_4d5f_8bfb_01c80bc6c179.slice/crio-6ac82ed337be18d402f1232309160b856c7a0a920dafb54cd33c23a210afc1fe WatchSource:0}: Error finding container 6ac82ed337be18d402f1232309160b856c7a0a920dafb54cd33c23a210afc1fe: Status 404 returned error can't find the container with id 6ac82ed337be18d402f1232309160b856c7a0a920dafb54cd33c23a210afc1fe
Nov 03 20:50:55.356874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:55.356839    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg" event={"ID":"6c684363-f63c-41d5-af34-d3232e1feef0","Type":"ContainerStarted","Data":"4dcb347691ccdfb83673ac198b67211bc7438f4d376812d2fb995f2598ee4d21"}
Nov 03 20:50:55.357064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:55.356895    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg" event={"ID":"6c684363-f63c-41d5-af34-d3232e1feef0","Type":"ContainerStarted","Data":"e2e8e192354bf80fd637d8f979985abe1150164475410ad14261201342d0f188"}
Nov 03 20:50:55.364279 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:55.364231    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" event={"ID":"784d401d-fb0d-4d5f-8bfb-01c80bc6c179","Type":"ContainerStarted","Data":"f83a45737dcae38b5f46924525ec4286353435f642d27402b88b4391ba670887"}
Nov 03 20:50:55.364468 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:55.364286    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" event={"ID":"784d401d-fb0d-4d5f-8bfb-01c80bc6c179","Type":"ContainerStarted","Data":"6ac82ed337be18d402f1232309160b856c7a0a920dafb54cd33c23a210afc1fe"}
Nov 03 20:50:55.464721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:55.464668    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg" podStartSLOduration=2.464653719 podStartE2EDuration="2.464653719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:55.464396068 +0000 UTC m=+1295.468354812" watchObservedRunningTime="2026-11-03 20:50:55.464653719 +0000 UTC m=+1295.468612443"
Nov 03 20:50:55.501863 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:55.501822    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:50:55.573256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:55.573191    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podStartSLOduration=1.5731764990000001 podStartE2EDuration="1.573176499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:55.571579351 +0000 UTC m=+1295.575538094" watchObservedRunningTime="2026-11-03 20:50:55.573176499 +0000 UTC m=+1295.577135223"
Nov 03 20:50:56.258985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:56.258951    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:56.374921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:56.374879    5196 generic.go:356] "Generic (PLEG): container finished" podID="6c684363-f63c-41d5-af34-d3232e1feef0" containerID="4dcb347691ccdfb83673ac198b67211bc7438f4d376812d2fb995f2598ee4d21" exitCode=0
Nov 03 20:50:56.375151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:56.374959    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg" event={"ID":"6c684363-f63c-41d5-af34-d3232e1feef0","Type":"ContainerDied","Data":"4dcb347691ccdfb83673ac198b67211bc7438f4d376812d2fb995f2598ee4d21"}
Nov 03 20:50:56.919837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:56.919798    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:57.063669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.063638    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:57.564251 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.564219    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:57.667357 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.667263    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c684363-f63c-41d5-af34-d3232e1feef0-kube-api-access-bp6z2\" (UniqueName: \"kubernetes.io/projected/6c684363-f63c-41d5-af34-d3232e1feef0-kube-api-access-bp6z2\") pod \"6c684363-f63c-41d5-af34-d3232e1feef0\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") "
Nov 03 20:50:57.667528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.667433    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-openshift-ca\") pod \"6c684363-f63c-41d5-af34-d3232e1feef0\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") "
Nov 03 20:50:57.667592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.667559    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-xks-ca\") pod \"6c684363-f63c-41d5-af34-d3232e1feef0\" (UID: \"6c684363-f63c-41d5-af34-d3232e1feef0\") "
Nov 03 20:50:57.667821 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.667786    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-openshift-ca" pod "6c684363-f63c-41d5-af34-d3232e1feef0" (UID: "6c684363-f63c-41d5-af34-d3232e1feef0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:57.668074 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.668049    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:57.669996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.669968    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c684363-f63c-41d5-af34-d3232e1feef0-kube-api-access-bp6z2" pod "6c684363-f63c-41d5-af34-d3232e1feef0" (UID: "6c684363-f63c-41d5-af34-d3232e1feef0"). InnerVolumeSpecName "kube-api-access-bp6z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:57.670083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.669999    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-xks-ca" pod "6c684363-f63c-41d5-af34-d3232e1feef0" (UID: "6c684363-f63c-41d5-af34-d3232e1feef0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:50:57.769187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.769142    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c684363-f63c-41d5-af34-d3232e1feef0-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:57.769187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:57.769179    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp6z2\" (UniqueName: \"kubernetes.io/projected/6c684363-f63c-41d5-af34-d3232e1feef0-kube-api-access-bp6z2\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:58.046717 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:58.046622    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"]
Nov 03 20:50:58.051479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:58.051443    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ea385e-rhw6v"]
Nov 03 20:50:58.112654 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:58.112616    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25841a65-a68a-450b-b6f2-1b54b45e8908" path="/var/lib/kubelet/pods/25841a65-a68a-450b-b6f2-1b54b45e8908/volumes"
Nov 03 20:50:58.395332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:58.395295    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"
Nov 03 20:50:58.395332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:58.395307    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg" event={"ID":"6c684363-f63c-41d5-af34-d3232e1feef0","Type":"ContainerDied","Data":"e2e8e192354bf80fd637d8f979985abe1150164475410ad14261201342d0f188"}
Nov 03 20:50:58.395554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:50:58.395355    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e8e192354bf80fd637d8f979985abe1150164475410ad14261201342d0f188"
Nov 03 20:51:00.643890 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:00.643839    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:00.999811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:00.999703    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:04.501889 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:04.501851    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:51:04.515920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:04.515891    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:51:05.467978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:05.467946    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:51:07.399578 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:07.399536    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:09.104121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:09.077033    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"]
Nov 03 20:51:09.125745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:09.125700    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f97cb6-lx58q"]
Nov 03 20:51:10.113706 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:10.113661    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653a316f-049d-47e8-a03e-a36ed0869755" path="/var/lib/kubelet/pods/653a316f-049d-47e8-a03e-a36ed0869755/volumes"
Nov 03 20:51:10.680983 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:10.680949    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:13.890953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.890917    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"]
Nov 03 20:51:13.891720 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.891702    5196 cpu_manager.go:414] "containerMap: removing container" podUID="6c684363-f63c-41d5-af34-d3232e1feef0" containerName="revoke-keys"
Nov 03 20:51:13.891783 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.891721    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c684363-f63c-41d5-af34-d3232e1feef0" containerName="revoke-keys"
Nov 03 20:51:13.891851 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.891840    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c684363-f63c-41d5-af34-d3232e1feef0" containerName="revoke-keys"
Nov 03 20:51:13.896726 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.896703    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:13.928000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.927967    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:13.928214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.928017    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwfk\" (UniqueName: \"kubernetes.io/projected/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-kube-api-access-jnwfk\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:13.928214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.928179    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:13.961156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:13.961121    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"]
Nov 03 20:51:14.002190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.002145    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:51:14.002387 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.002298    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:51:14.029617 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.029564    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:14.029832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.029635    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwfk\" (UniqueName: \"kubernetes.io/projected/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-kube-api-access-jnwfk\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:14.029832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.029693    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:14.030275 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.030253    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:14.032342 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.032318    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:14.085321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.085285    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwfk\" (UniqueName: \"kubernetes.io/projected/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-kube-api-access-jnwfk\") pod \"maas-api-revoke-sub-8a6e0fa4-6xgwf\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:14.212047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.211939    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:14.601845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:14.601802    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"]
Nov 03 20:51:14.606902 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:14.606873    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fcf07a_e119_43a5_9fdd_1d922e1c476e.slice/crio-6874593ba882710fc9d4cd660b704102ab2b6f6656db925fbc1e3b55a159637a WatchSource:0}: Error finding container 6874593ba882710fc9d4cd660b704102ab2b6f6656db925fbc1e3b55a159637a: Status 404 returned error can't find the container with id 6874593ba882710fc9d4cd660b704102ab2b6f6656db925fbc1e3b55a159637a
Nov 03 20:51:15.523630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:15.523574    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:15.592260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:15.592213    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf" event={"ID":"c5fcf07a-e119-43a5-9fdd-1d922e1c476e","Type":"ContainerStarted","Data":"7432c2eb6baf548f243d754b00e261671e4986b3a093b2abfaa5063d69f17c0c"}
Nov 03 20:51:15.592448 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:15.592340    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf" event={"ID":"c5fcf07a-e119-43a5-9fdd-1d922e1c476e","Type":"ContainerStarted","Data":"6874593ba882710fc9d4cd660b704102ab2b6f6656db925fbc1e3b55a159637a"}
Nov 03 20:51:15.701886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:15.701827    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf" podStartSLOduration=2.701813214 podStartE2EDuration="2.701813214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:15.698973241 +0000 UTC m=+1315.702931991" watchObservedRunningTime="2026-11-03 20:51:15.701813214 +0000 UTC m=+1315.705771937"
Nov 03 20:51:16.601861 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:16.601825    5196 generic.go:356] "Generic (PLEG): container finished" podID="c5fcf07a-e119-43a5-9fdd-1d922e1c476e" containerID="7432c2eb6baf548f243d754b00e261671e4986b3a093b2abfaa5063d69f17c0c" exitCode=0
Nov 03 20:51:16.602396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:16.601908    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf" event={"ID":"c5fcf07a-e119-43a5-9fdd-1d922e1c476e","Type":"ContainerDied","Data":"7432c2eb6baf548f243d754b00e261671e4986b3a093b2abfaa5063d69f17c0c"}
Nov 03 20:51:17.614455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.614427    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"]
Nov 03 20:51:17.619276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.619258    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.672907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.672873    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"]
Nov 03 20:51:17.679195 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.679165    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.679385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.679326    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4spt\" (UniqueName: \"kubernetes.io/projected/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-kube-api-access-x4spt\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.679385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.679378    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.780594 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.780564    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.780789 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.780673    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4spt\" (UniqueName: \"kubernetes.io/projected/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-kube-api-access-x4spt\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.780789 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.780713    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.781627 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.781599    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.783763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.783737    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.819715 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.819691    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:17.834649 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.834618    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4spt\" (UniqueName: \"kubernetes.io/projected/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-kube-api-access-x4spt\") pod \"maas-api-revoke-sub-42c1d91e-k4bt5\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.881998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.881916    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-kube-api-access-jnwfk\" (UniqueName: \"kubernetes.io/projected/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-kube-api-access-jnwfk\") pod \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") "
Nov 03 20:51:17.882214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.882149    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-openshift-ca\") pod \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") "
Nov 03 20:51:17.882214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.882195    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-xks-ca\") pod \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\" (UID: \"c5fcf07a-e119-43a5-9fdd-1d922e1c476e\") "
Nov 03 20:51:17.882597 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.882567    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-openshift-ca" pod "c5fcf07a-e119-43a5-9fdd-1d922e1c476e" (UID: "c5fcf07a-e119-43a5-9fdd-1d922e1c476e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:17.883045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.883020    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:17.884474 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.884447    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-xks-ca" pod "c5fcf07a-e119-43a5-9fdd-1d922e1c476e" (UID: "c5fcf07a-e119-43a5-9fdd-1d922e1c476e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:17.884566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.884447    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-kube-api-access-jnwfk" pod "c5fcf07a-e119-43a5-9fdd-1d922e1c476e" (UID: "c5fcf07a-e119-43a5-9fdd-1d922e1c476e"). InnerVolumeSpecName "kube-api-access-jnwfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:17.936497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.936462    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:17.984526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.984487    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:17.984526 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:17.984521    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwfk\" (UniqueName: \"kubernetes.io/projected/c5fcf07a-e119-43a5-9fdd-1d922e1c476e-kube-api-access-jnwfk\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:18.321677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:18.320879    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"]
Nov 03 20:51:18.487360 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:51:18.487323    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:51:18.621811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:18.621717    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5" event={"ID":"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7","Type":"ContainerStarted","Data":"7c55c31da3090fce2c6d1c89dfced6c125d5dcfb982b6193427fe7790f06783a"}
Nov 03 20:51:18.623391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:18.623364    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf" event={"ID":"c5fcf07a-e119-43a5-9fdd-1d922e1c476e","Type":"ContainerDied","Data":"6874593ba882710fc9d4cd660b704102ab2b6f6656db925fbc1e3b55a159637a"}
Nov 03 20:51:18.623500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:18.623398    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"
Nov 03 20:51:18.623500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:18.623415    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6874593ba882710fc9d4cd660b704102ab2b6f6656db925fbc1e3b55a159637a"
Nov 03 20:51:19.633485 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:19.633449    5196 generic.go:356] "Generic (PLEG): container finished" podID="492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7" containerID="dddd4922e8997b3ecb9bc909869c979735f14f65a02fdffa7a6e25d7a534d94d" exitCode=0
Nov 03 20:51:19.633886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:19.633530    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5" event={"ID":"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7","Type":"ContainerDied","Data":"dddd4922e8997b3ecb9bc909869c979735f14f65a02fdffa7a6e25d7a534d94d"}
Nov 03 20:51:20.048033 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.047953    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"]
Nov 03 20:51:20.052539 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.052508    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b18abc3-tn6s5"]
Nov 03 20:51:20.111459 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.111430    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d6b4ff-8175-454c-9b4a-0a86b435dea5" path="/var/lib/kubelet/pods/56d6b4ff-8175-454c-9b4a-0a86b435dea5/volumes"
Nov 03 20:51:20.263768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.263736    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:20.816186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.816159    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:20.923010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.922923    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-xks-ca\") pod \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") "
Nov 03 20:51:20.923283 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.923263    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-openshift-ca\") pod \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") "
Nov 03 20:51:20.923400 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.923387    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-kube-api-access-x4spt\" (UniqueName: \"kubernetes.io/projected/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-kube-api-access-x4spt\") pod \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\" (UID: \"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7\") "
Nov 03 20:51:20.924915 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.924883    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-openshift-ca" pod "492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7" (UID: "492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:20.925841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.925816    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-xks-ca" pod "492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7" (UID: "492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:20.927609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:20.927570    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-kube-api-access-x4spt" pod "492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7" (UID: "492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7"). InnerVolumeSpecName "kube-api-access-x4spt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:21.025932 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:21.025889    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:21.025932 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:21.025928    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:21.025932 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:21.025941    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4spt\" (UniqueName: \"kubernetes.io/projected/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7-kube-api-access-x4spt\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:21.651053 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:21.651028    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"
Nov 03 20:51:21.651281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:21.651025    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5" event={"ID":"492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7","Type":"ContainerDied","Data":"7c55c31da3090fce2c6d1c89dfced6c125d5dcfb982b6193427fe7790f06783a"}
Nov 03 20:51:21.651281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:21.651157    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c55c31da3090fce2c6d1c89dfced6c125d5dcfb982b6193427fe7790f06783a"
Nov 03 20:51:24.559180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.559142    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"]
Nov 03 20:51:24.559909 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.559892    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c5fcf07a-e119-43a5-9fdd-1d922e1c476e" containerName="revoke-keys"
Nov 03 20:51:24.559950 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.559913    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fcf07a-e119-43a5-9fdd-1d922e1c476e" containerName="revoke-keys"
Nov 03 20:51:24.559985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.559954    5196 cpu_manager.go:414] "containerMap: removing container" podUID="492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7" containerName="revoke-keys"
Nov 03 20:51:24.559985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.559960    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7" containerName="revoke-keys"
Nov 03 20:51:24.560126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.560081    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7" containerName="revoke-keys"
Nov 03 20:51:24.560126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.560120    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fcf07a-e119-43a5-9fdd-1d922e1c476e" containerName="revoke-keys"
Nov 03 20:51:24.564189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.564166    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.658031 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.657994    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"]
Nov 03 20:51:24.673646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.673612    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.673646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.673654    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.673868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.673691    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.673868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.673825    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.673955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.673907    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.674079 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.674061    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9chb\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-kube-api-access-q9chb\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.674190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.674144    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.674190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.674180    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.674288 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.674272    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775481 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775439    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775481 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775486    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775513    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775548    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775735 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775598    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775914 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775739    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9chb\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-kube-api-access-q9chb\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775914 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775775    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775914 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775807    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.775914 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775892    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.776165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.775919    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.776165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.776005    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.776303 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.776267    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.776360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.776272    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.776419 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.776357    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.778152 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.778131    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.778388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.778368    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.830064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.830026    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.830227 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.830181    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9chb\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-kube-api-access-q9chb\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:24.881088 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:24.881046    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:25.015046 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.015006    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:25.322424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.322395    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"]
Nov 03 20:51:25.325012 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:25.324978    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa307edc_bb9a_4d62_9c3b_17388df1bab9.slice/crio-f5d736f17dfc5f916cc765028db9227deaf25edeaa5c9c5599de4821c40fcd42 WatchSource:0}: Error finding container f5d736f17dfc5f916cc765028db9227deaf25edeaa5c9c5599de4821c40fcd42: Status 404 returned error can't find the container with id f5d736f17dfc5f916cc765028db9227deaf25edeaa5c9c5599de4821c40fcd42
Nov 03 20:51:25.327744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.327709    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:25.327874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.327780    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:25.327874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.327827    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:25.688927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.688823    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" event={"ID":"fa307edc-bb9a-4d62-9c3b-17388df1bab9","Type":"ContainerStarted","Data":"c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d"}
Nov 03 20:51:25.688927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.688895    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" event={"ID":"fa307edc-bb9a-4d62-9c3b-17388df1bab9","Type":"ContainerStarted","Data":"f5d736f17dfc5f916cc765028db9227deaf25edeaa5c9c5599de4821c40fcd42"}
Nov 03 20:51:25.801514 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.801456    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podStartSLOduration=1.801440962 podStartE2EDuration="1.801440962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:25.799123784 +0000 UTC m=+1325.803082525" watchObservedRunningTime="2026-11-03 20:51:25.801440962 +0000 UTC m=+1325.805399686"
Nov 03 20:51:25.882172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:25.882139    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:26.855126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.850214    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"]
Nov 03 20:51:26.863120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.861996    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:26.881480 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.881450    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"]
Nov 03 20:51:26.883413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.883384    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:26.883534 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.883441    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:26.886120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.886085    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:26.912744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.912715    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-41aa21-openshift-default-dockercfg-plbb2\""
Nov 03 20:51:26.931522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.931475    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"]
Nov 03 20:51:26.990205 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:26.990166    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"]
Nov 03 20:51:27.007483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007446    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.007679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007602    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.007679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007637    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56q9\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-kube-api-access-l56q9\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.007679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007667    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.007896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007682    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.007896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007824    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.007896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007849    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvmg\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-kube-api-access-zrvmg\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.007896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007870    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.008134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007929    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.008134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007962    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.008134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.007991    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.008134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.008042    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.008134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.008082    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.008134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.008131    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.008442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.008158    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.008442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.008178    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.008442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.008283    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290be761-fb7e-4715-9a49-856178eb262c-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.008442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.008308    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290be761-fb7e-4715-9a49-856178eb262c-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.109348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109254    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.109517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109373    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.109517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109400    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56q9\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-kube-api-access-l56q9\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.109517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109421    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.109517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109450    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.109517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109494    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.109775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109517    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvmg\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-kube-api-access-zrvmg\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.109775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109545    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.109775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109580    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.109775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109608    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.109775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109659    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.109775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109701    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.109775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109770    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109784    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109810    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109847    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109852    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109874    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109901    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.109989    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290be761-fb7e-4715-9a49-856178eb262c-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.110025    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290be761-fb7e-4715-9a49-856178eb262c-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.110162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.110040    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-certs\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.110614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.110230    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.110823 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.110799    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.111169 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.110975    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-data\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.111169 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.111142    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290be761-fb7e-4715-9a49-856178eb262c-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.111169 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.111146    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-credential-socket\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.111344 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.111281    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istiod-ca-cert\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.112800 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.112767    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.112902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.112778    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290be761-fb7e-4715-9a49-856178eb262c-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.112902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.112846    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-envoy\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.113019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.113002    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-podinfo\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.164733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.164694    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-token\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.165255 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.165229    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56q9\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-kube-api-access-l56q9\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.165824 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.165807    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvmg\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-kube-api-access-zrvmg\") pod \"e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") " pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.166351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.166321    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.181313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.181282    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:27.203855 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.203816    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:27.677293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.676174    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"]
Nov 03 20:51:27.679689 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.679655    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:27.679802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.679769    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:27.679802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.679788    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:27.679916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.679825    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:27.679916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.679840    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:27.679916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.679861    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:27.681155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.681133    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"]
Nov 03 20:51:27.710269 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.710236    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" event={"ID":"3eb104ce-df5c-489d-b5d8-2d6a41b8c218","Type":"ContainerStarted","Data":"9c735078b3585cd939247d075bef7abcede114b11ca3de2678ba9214209e77e6"}
Nov 03 20:51:27.711774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.711750    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" event={"ID":"290be761-fb7e-4715-9a49-856178eb262c","Type":"ContainerStarted","Data":"513578ea03bec75792ba5fbe29a8bc3f57a17cf49c7e7c994b608a802f9a1c69"}
Nov 03 20:51:27.881593 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.881552    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:27.882143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:27.881619    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:28.733725 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:28.733685    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" event={"ID":"290be761-fb7e-4715-9a49-856178eb262c","Type":"ContainerStarted","Data":"2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c"}
Nov 03 20:51:28.753119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:28.742024    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" event={"ID":"3eb104ce-df5c-489d-b5d8-2d6a41b8c218","Type":"ContainerStarted","Data":"64d87e957d50f1882043d3a5f426722366ee2feead604bf78eeed38ad827c214"}
Nov 03 20:51:28.888125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:28.876679    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podStartSLOduration=2.876656641 podStartE2EDuration="2.876656641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:28.849623 +0000 UTC m=+1328.853581724" watchObservedRunningTime="2026-11-03 20:51:28.876656641 +0000 UTC m=+1328.880615366"
Nov 03 20:51:28.888125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:28.886752    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:28.888125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:28.886805    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:29.198123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.187268    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:29.198123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.189402    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:51:29.198123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.189469    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 20:51:29.217122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.205398    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:29.217122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.207490    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:51:29.217122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.207565    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 20:51:29.304118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.301928    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podStartSLOduration=3.301902166 podStartE2EDuration="3.301902166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:28.987670104 +0000 UTC m=+1328.991628854" watchObservedRunningTime="2026-11-03 20:51:29.301902166 +0000 UTC m=+1329.305860890"
Nov 03 20:51:29.304118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.303306    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:51:29.888123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.881441    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:29.888123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:29.881502    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:30.212181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.211940    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"]
Nov 03 20:51:30.272996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.234255    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:51:30.272996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.234325    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 20:51:30.272996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.234823    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:51:30.272996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.234869    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 20:51:30.272996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.271905    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:30.350120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.345283    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.393130 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.393049    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"]
Nov 03 20:51:30.409248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.404484    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:51:30.409248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.405206    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:51:30.431128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.428325    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.431128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.428437    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4n2l\" (UniqueName: \"kubernetes.io/projected/ef96d9d8-7a54-4375-93ae-46fc0916677a-kube-api-access-c4n2l\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.431128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.428507    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.580500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.577598    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.580500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.577869    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.580500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.577974    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4n2l\" (UniqueName: \"kubernetes.io/projected/ef96d9d8-7a54-4375-93ae-46fc0916677a-kube-api-access-c4n2l\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.592054 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.582494    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.611260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.603999    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.668057 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.668020    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4n2l\" (UniqueName: \"kubernetes.io/projected/ef96d9d8-7a54-4375-93ae-46fc0916677a-kube-api-access-c4n2l\") pod \"maas-api-revoke-sub-72d4a7fa-2gbf9\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:30.721976 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.721345    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:30.931124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.916233    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:30.931124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.923262    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:30.951120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.949808    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:30.951120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.949870    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:31.000274 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:30.997525    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:31.228121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.228026    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.228333 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.228121    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 20:51:31.305122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.287271    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.305122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.287334    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 20:51:31.351121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.350247    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:31.376122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.372763    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:31.437184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.436961    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:31.458465 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.458421    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": context deadline exceeded"
Nov 03 20:51:31.584400 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.584347    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 20:51:31.585155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.585073    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 20:51:31.735559 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.728206    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 03 20:51:31.735559 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.728725    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:51:31.893160 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.886205    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.893160 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.886265    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:31.940313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.940252    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": context deadline exceeded"
Nov 03 20:51:31.967957 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:31.967910    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:51:32.282118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.281519    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/health\": context deadline exceeded"
Nov 03 20:51:32.338121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.319725    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:51:32.338121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.319791    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 20:51:32.338121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.328983    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:51:32.338121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.329038    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 20:51:32.364513 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.364468    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:32.417122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.394341    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 03 20:51:32.417122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.395159    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded"
Nov 03 20:51:32.447284 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.447225    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:32.520126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.518354    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:32.592307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.592263    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:32.627121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.602076    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:32.737516 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.737454    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:32.915150 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:32.915081    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.032237 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.032183    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.100123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.072212    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:33.127117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.108609    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:33.127117 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:51:33.112646    5196 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1s"
Nov 03 20:51:33.127117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.112892    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:51:33.140264 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.140230    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:51:33.140907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.140890    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:51:33.141121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.141087    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:51:33.145240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.145218    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:51:33.173484 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.173366    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:51:33.229242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.222170    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:51:33.229242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.222234    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 20:51:33.303315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.296565    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:51:33.303315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.296629    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 20:51:33.372943 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.372899    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.409149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.406276    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.426137 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.409410    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 03 20:51:33.426137 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.410799    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:51:33.450592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.436694    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.512248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.512199    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.523270 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.521296    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.610551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.610520    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:51:33.666406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.666373    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:51:33.709256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.691192    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.709256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.691436    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 20:51:33.718185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.717930    5196 trace.go:236] Trace[2051769776]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-768646756f-f86vn" (03-Nov-2026 20:51:32.413) (total time: 1304ms):
Nov 03 20:51:33.718185 ip-10-0-128-49 kubenswrapper[5196]: Trace[2051769776]: [1.304671455s] [1.304671455s] END
Nov 03 20:51:33.758376 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.748181    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.811770 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.811728    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:33.931768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.930068    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:51:33.960211 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.954061    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:51:33.990121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.961820    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:33.990121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.961891    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:33.994115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:33.992369    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:51:34.013409 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:34.013374    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef96d9d8_7a54_4375_93ae_46fc0916677a.slice/crio-2745755a755342ddc0ede360f92acad4defac70fb676ad9502dbcb2de101e1d5 WatchSource:0}: Error finding container 2745755a755342ddc0ede360f92acad4defac70fb676ad9502dbcb2de101e1d5: Status 404 returned error can't find the container with id 2745755a755342ddc0ede360f92acad4defac70fb676ad9502dbcb2de101e1d5
Nov 03 20:51:34.120127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.065178    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:34.120127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.078408    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"]
Nov 03 20:51:34.267182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.222174    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:51:34.267182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.222254    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 20:51:34.330119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.321620    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:51:34.330119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.321676    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 20:51:34.330119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.321946    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:51:34.330119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.322010    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:51:34.330119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.322052    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:51:34.330119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.322083    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:51:34.330119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.329865    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"]
Nov 03 20:51:34.475662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.475623    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:51:34.475944 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.475926    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:51:34.476128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.476083    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:51:34.477267 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.477248    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.516870 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.515569    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"]
Nov 03 20:51:34.717071 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.716941    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cgnq\" (UniqueName: \"kubernetes.io/projected/49d18462-f811-4f8a-8c1f-3b894975231b-kube-api-access-7cgnq\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.717484 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.717404    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.717789 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.717752    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.816123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.815992    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9" event={"ID":"ef96d9d8-7a54-4375-93ae-46fc0916677a","Type":"ContainerStarted","Data":"2745755a755342ddc0ede360f92acad4defac70fb676ad9502dbcb2de101e1d5"}
Nov 03 20:51:34.821826 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.821781    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgnq\" (UniqueName: \"kubernetes.io/projected/49d18462-f811-4f8a-8c1f-3b894975231b-kube-api-access-7cgnq\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.830586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.830537    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.831335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.831301    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.838126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.834393    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.922303 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.922048    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 20:51:34.922573 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.922541    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 20:51:34.941824 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.931067    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:34.966120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:34.962864    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgnq\" (UniqueName: \"kubernetes.io/projected/49d18462-f811-4f8a-8c1f-3b894975231b-kube-api-access-7cgnq\") pod \"maas-api-revoke-sub-f40841e4-r74vw\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:35.193611 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:35.193570    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:51:35.206029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:35.205979    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 20:51:35.209012 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:35.208976    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:51:35.209295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:35.209268    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 20:51:35.209622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:35.209602    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:36.085740 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.062976    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9" event={"ID":"ef96d9d8-7a54-4375-93ae-46fc0916677a","Type":"ContainerStarted","Data":"2e2b6c7c1b8a2201fe29209c4dcac30adc8e6a4365a083f44f364c4254854f37"}
Nov 03 20:51:36.127979 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.121643    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:36.134457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.134425    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:36.136262 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.136240    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:36.138216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.138198    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 20:51:36.172750 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.172704    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"]
Nov 03 20:51:36.186492 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.186458    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:51:36.186755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.186726    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 20:51:36.192189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.190856    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.239875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.236204    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"]
Nov 03 20:51:36.239875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.237139    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:51:36.239875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.237194    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 20:51:36.320891 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.315781    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9" podStartSLOduration=6.315759063 podStartE2EDuration="6.315759063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:36.308163564 +0000 UTC m=+1336.312122299" watchObservedRunningTime="2026-11-03 20:51:36.315759063 +0000 UTC m=+1336.319717789"
Nov 03 20:51:36.367753 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.367699    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.367930 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.367775    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6kq\" (UniqueName: \"kubernetes.io/projected/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-kube-api-access-2r6kq\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.368013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.367968    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.420383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.420350    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"]
Nov 03 20:51:36.462120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.461889    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"]
Nov 03 20:51:36.469118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.469041    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6kq\" (UniqueName: \"kubernetes.io/projected/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-kube-api-access-2r6kq\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.470065 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.469413    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.470065 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.469595    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.472366 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.471236    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.473140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.472722    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.492057 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.491993    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.509237 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.504287    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"]
Nov 03 20:51:36.537056 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.537010    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6kq\" (UniqueName: \"kubernetes.io/projected/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-kube-api-access-2r6kq\") pod \"maas-api-revoke-sub-046108f4-dqq7s\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.571501 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.571410    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5kwx\" (UniqueName: \"kubernetes.io/projected/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-kube-api-access-q5kwx\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.571847 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.571823    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.572015 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.571961    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.575371 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.575337    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:36.674334 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.674291    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.674631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.674606    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.676478 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.676328    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.677029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.676925    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5kwx\" (UniqueName: \"kubernetes.io/projected/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-kube-api-access-q5kwx\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.677762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.677736    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.749534 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.749463    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5kwx\" (UniqueName: \"kubernetes.io/projected/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-kube-api-access-q5kwx\") pod \"maas-api-revoke-sub-81402c5e-khb4w\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:36.972289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:36.972249    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:37.026933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.025967    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"]
Nov 03 20:51:37.030016 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:37.029986    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba6a943_fec3_4cee_8ec5_8687bb2308bb.slice/crio-591805fa7ed332a648c468135c5217a4af24ba4ed7ec416d28e058ef023bd587 WatchSource:0}: Error finding container 591805fa7ed332a648c468135c5217a4af24ba4ed7ec416d28e058ef023bd587: Status 404 returned error can't find the container with id 591805fa7ed332a648c468135c5217a4af24ba4ed7ec416d28e058ef023bd587
Nov 03 20:51:37.064325 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.064287    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"]
Nov 03 20:51:37.079028 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.078991    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211bcda9-db97d"]
Nov 03 20:51:37.104139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.098836    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s" event={"ID":"2ba6a943-fec3-4cee-8ec5-8687bb2308bb","Type":"ContainerStarted","Data":"591805fa7ed332a648c468135c5217a4af24ba4ed7ec416d28e058ef023bd587"}
Nov 03 20:51:37.104139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.100751    5196 generic.go:356] "Generic (PLEG): container finished" podID="ef96d9d8-7a54-4375-93ae-46fc0916677a" containerID="2e2b6c7c1b8a2201fe29209c4dcac30adc8e6a4365a083f44f364c4254854f37" exitCode=0
Nov 03 20:51:37.104139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.100810    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9" event={"ID":"ef96d9d8-7a54-4375-93ae-46fc0916677a","Type":"ContainerDied","Data":"2e2b6c7c1b8a2201fe29209c4dcac30adc8e6a4365a083f44f364c4254854f37"}
Nov 03 20:51:37.110153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.108273    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw" event={"ID":"49d18462-f811-4f8a-8c1f-3b894975231b","Type":"ContainerStarted","Data":"660a39eeb4cba88d0b8448a5031144fc974d64ffcaf1597dbc1213ad75c777f3"}
Nov 03 20:51:37.110153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.108320    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw" event={"ID":"49d18462-f811-4f8a-8c1f-3b894975231b","Type":"ContainerStarted","Data":"cb26f646072bf84cb21867a8ad46ee883f89c419e810cccbe81f24808d0ccf03"}
Nov 03 20:51:37.318585 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.318526    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw" podStartSLOduration=3.318511875 podStartE2EDuration="3.318511875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:37.317192983 +0000 UTC m=+1337.321151726" watchObservedRunningTime="2026-11-03 20:51:37.318511875 +0000 UTC m=+1337.322470598"
Nov 03 20:51:37.370056 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.370013    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"]
Nov 03 20:51:37.664609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.664573    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:37.665122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.665029    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:37.666549 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:37.666525    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:51:38.113396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.113365    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef277745-615c-4e9e-9c98-7c1f2e7455cf" path="/var/lib/kubelet/pods/ef277745-615c-4e9e-9c98-7c1f2e7455cf/volumes"
Nov 03 20:51:38.123361 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.123332    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w" event={"ID":"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909","Type":"ContainerStarted","Data":"f9e60e1e91337bbc4e30437f7ee7f292b1419c9668a9cd67d6f81b98fc50ec3c"}
Nov 03 20:51:38.123493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.123385    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w" event={"ID":"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909","Type":"ContainerStarted","Data":"048eb4a470e56a101386bccdfa60e8c655b19c3d27319f0943463845fc94a522"}
Nov 03 20:51:38.125373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.125347    5196 generic.go:356] "Generic (PLEG): container finished" podID="49d18462-f811-4f8a-8c1f-3b894975231b" containerID="660a39eeb4cba88d0b8448a5031144fc974d64ffcaf1597dbc1213ad75c777f3" exitCode=0
Nov 03 20:51:38.125488 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.125441    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw" event={"ID":"49d18462-f811-4f8a-8c1f-3b894975231b","Type":"ContainerDied","Data":"660a39eeb4cba88d0b8448a5031144fc974d64ffcaf1597dbc1213ad75c777f3"}
Nov 03 20:51:38.127347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.127325    5196 generic.go:356] "Generic (PLEG): container finished" podID="2ba6a943-fec3-4cee-8ec5-8687bb2308bb" containerID="80be9e584cd4cf807850bd52e461c0374e84a315a893f774b6d54e52eba663dd" exitCode=0
Nov 03 20:51:38.127453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.127382    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s" event={"ID":"2ba6a943-fec3-4cee-8ec5-8687bb2308bb","Type":"ContainerDied","Data":"80be9e584cd4cf807850bd52e461c0374e84a315a893f774b6d54e52eba663dd"}
Nov 03 20:51:38.248172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.248132    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:51:38.248372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.248216    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:38.249539 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.249470    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w" podStartSLOduration=2.249451923 podStartE2EDuration="2.249451923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:38.230378405 +0000 UTC m=+1338.234337148" watchObservedRunningTime="2026-11-03 20:51:38.249451923 +0000 UTC m=+1338.253410649"
Nov 03 20:51:38.333435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.333402    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:38.417458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.417374    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-xks-ca\") pod \"ef96d9d8-7a54-4375-93ae-46fc0916677a\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") "
Nov 03 20:51:38.417622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.417495    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef96d9d8-7a54-4375-93ae-46fc0916677a-kube-api-access-c4n2l\" (UniqueName: \"kubernetes.io/projected/ef96d9d8-7a54-4375-93ae-46fc0916677a-kube-api-access-c4n2l\") pod \"ef96d9d8-7a54-4375-93ae-46fc0916677a\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") "
Nov 03 20:51:38.417681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.417630    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-openshift-ca\") pod \"ef96d9d8-7a54-4375-93ae-46fc0916677a\" (UID: \"ef96d9d8-7a54-4375-93ae-46fc0916677a\") "
Nov 03 20:51:38.418007 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.417978    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-openshift-ca" pod "ef96d9d8-7a54-4375-93ae-46fc0916677a" (UID: "ef96d9d8-7a54-4375-93ae-46fc0916677a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:38.418368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.418348    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:38.419838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.419810    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-xks-ca" pod "ef96d9d8-7a54-4375-93ae-46fc0916677a" (UID: "ef96d9d8-7a54-4375-93ae-46fc0916677a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:38.419838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.419819    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef96d9d8-7a54-4375-93ae-46fc0916677a-kube-api-access-c4n2l" pod "ef96d9d8-7a54-4375-93ae-46fc0916677a" (UID: "ef96d9d8-7a54-4375-93ae-46fc0916677a"). InnerVolumeSpecName "kube-api-access-c4n2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:38.520034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.519996    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef96d9d8-7a54-4375-93ae-46fc0916677a-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:38.520034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.520031    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4n2l\" (UniqueName: \"kubernetes.io/projected/ef96d9d8-7a54-4375-93ae-46fc0916677a-kube-api-access-c4n2l\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:38.992005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.991971    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:38.994272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.992685    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:38.997300 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:38.997276    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 20:51:39.139935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.139902    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"
Nov 03 20:51:39.140197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.139902    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9" event={"ID":"ef96d9d8-7a54-4375-93ae-46fc0916677a","Type":"ContainerDied","Data":"2745755a755342ddc0ede360f92acad4defac70fb676ad9502dbcb2de101e1d5"}
Nov 03 20:51:39.140197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.140031    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2745755a755342ddc0ede360f92acad4defac70fb676ad9502dbcb2de101e1d5"
Nov 03 20:51:39.142166 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.142139    5196 generic.go:356] "Generic (PLEG): container finished" podID="b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909" containerID="f9e60e1e91337bbc4e30437f7ee7f292b1419c9668a9cd67d6f81b98fc50ec3c" exitCode=0
Nov 03 20:51:39.142308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.142267    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w" event={"ID":"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909","Type":"ContainerDied","Data":"f9e60e1e91337bbc4e30437f7ee7f292b1419c9668a9cd67d6f81b98fc50ec3c"}
Nov 03 20:51:39.348595 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.348562    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:39.397086 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.397055    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:39.435394 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.435363    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-xks-ca\") pod \"49d18462-f811-4f8a-8c1f-3b894975231b\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") "
Nov 03 20:51:39.435563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.435420    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-openshift-ca\") pod \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") "
Nov 03 20:51:39.435563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.435439    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-openshift-ca\") pod \"49d18462-f811-4f8a-8c1f-3b894975231b\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") "
Nov 03 20:51:39.435563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.435462    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-kube-api-access-2r6kq\" (UniqueName: \"kubernetes.io/projected/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-kube-api-access-2r6kq\") pod \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") "
Nov 03 20:51:39.435563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.435480    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d18462-f811-4f8a-8c1f-3b894975231b-kube-api-access-7cgnq\" (UniqueName: \"kubernetes.io/projected/49d18462-f811-4f8a-8c1f-3b894975231b-kube-api-access-7cgnq\") pod \"49d18462-f811-4f8a-8c1f-3b894975231b\" (UID: \"49d18462-f811-4f8a-8c1f-3b894975231b\") "
Nov 03 20:51:39.435563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.435514    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-xks-ca\") pod \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\" (UID: \"2ba6a943-fec3-4cee-8ec5-8687bb2308bb\") "
Nov 03 20:51:39.435955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.435922    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-openshift-ca" pod "49d18462-f811-4f8a-8c1f-3b894975231b" (UID: "49d18462-f811-4f8a-8c1f-3b894975231b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:39.436386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.436363    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:39.436763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.436740    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-openshift-ca" pod "2ba6a943-fec3-4cee-8ec5-8687bb2308bb" (UID: "2ba6a943-fec3-4cee-8ec5-8687bb2308bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:39.438027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.437998    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-xks-ca" pod "49d18462-f811-4f8a-8c1f-3b894975231b" (UID: "49d18462-f811-4f8a-8c1f-3b894975231b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:39.438331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.438300    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-xks-ca" pod "2ba6a943-fec3-4cee-8ec5-8687bb2308bb" (UID: "2ba6a943-fec3-4cee-8ec5-8687bb2308bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:39.438602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.438579    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-kube-api-access-2r6kq" pod "2ba6a943-fec3-4cee-8ec5-8687bb2308bb" (UID: "2ba6a943-fec3-4cee-8ec5-8687bb2308bb"). InnerVolumeSpecName "kube-api-access-2r6kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:39.438834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.438821    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d18462-f811-4f8a-8c1f-3b894975231b-kube-api-access-7cgnq" pod "49d18462-f811-4f8a-8c1f-3b894975231b" (UID: "49d18462-f811-4f8a-8c1f-3b894975231b"). InnerVolumeSpecName "kube-api-access-7cgnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:39.538310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.538229    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d18462-f811-4f8a-8c1f-3b894975231b-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:39.538310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.538261    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:39.538310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.538283    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r6kq\" (UniqueName: \"kubernetes.io/projected/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-kube-api-access-2r6kq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:39.538310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.538294    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cgnq\" (UniqueName: \"kubernetes.io/projected/49d18462-f811-4f8a-8c1f-3b894975231b-kube-api-access-7cgnq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:39.538310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.538303    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba6a943-fec3-4cee-8ec5-8687bb2308bb-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:39.680760 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.680723    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"]
Nov 03 20:51:39.681704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681674    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ef96d9d8-7a54-4375-93ae-46fc0916677a" containerName="revoke-keys"
Nov 03 20:51:39.681704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681697    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef96d9d8-7a54-4375-93ae-46fc0916677a" containerName="revoke-keys"
Nov 03 20:51:39.681875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681721    5196 cpu_manager.go:414] "containerMap: removing container" podUID="49d18462-f811-4f8a-8c1f-3b894975231b" containerName="revoke-keys"
Nov 03 20:51:39.681875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681727    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d18462-f811-4f8a-8c1f-3b894975231b" containerName="revoke-keys"
Nov 03 20:51:39.681875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681737    5196 cpu_manager.go:414] "containerMap: removing container" podUID="2ba6a943-fec3-4cee-8ec5-8687bb2308bb" containerName="revoke-keys"
Nov 03 20:51:39.681875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681742    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba6a943-fec3-4cee-8ec5-8687bb2308bb" containerName="revoke-keys"
Nov 03 20:51:39.682211 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681895    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba6a943-fec3-4cee-8ec5-8687bb2308bb" containerName="revoke-keys"
Nov 03 20:51:39.682211 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681914    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef96d9d8-7a54-4375-93ae-46fc0916677a" containerName="revoke-keys"
Nov 03 20:51:39.682211 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.681921    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d18462-f811-4f8a-8c1f-3b894975231b" containerName="revoke-keys"
Nov 03 20:51:39.686607 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.686587    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.740524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740489    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvzpv\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-kube-api-access-lvzpv\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.740748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740589    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.740748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740623    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.740748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740680    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.740949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740771    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.740949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740796    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207d53e6-6438-445a-a9c5-eda28ffed68f-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.740949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740837    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.740949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740879    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.741273 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.740989    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.743947 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.743879    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"]
Nov 03 20:51:39.842262 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842222    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842344    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvzpv\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-kube-api-access-lvzpv\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842411    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842442    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842488    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842558    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842582    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207d53e6-6438-445a-a9c5-eda28ffed68f-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842619    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842924 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842658    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.842924 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.842702    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.843174 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.843143    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.843282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.843152    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.843282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.843189    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.843745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.843715    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207d53e6-6438-445a-a9c5-eda28ffed68f-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.845197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.845178    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.845622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.845603    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.896762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.896725    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:39.896919 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:39.896839    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvzpv\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-kube-api-access-lvzpv\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:40.005384 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.005342    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:40.156810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.156785    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:51:40.156810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.156800    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s" event={"ID":"2ba6a943-fec3-4cee-8ec5-8687bb2308bb","Type":"ContainerDied","Data":"591805fa7ed332a648c468135c5217a4af24ba4ed7ec416d28e058ef023bd587"}
Nov 03 20:51:40.157060 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.156853    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="591805fa7ed332a648c468135c5217a4af24ba4ed7ec416d28e058ef023bd587"
Nov 03 20:51:40.158661 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.158637    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:51:40.158817 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.158669    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw" event={"ID":"49d18462-f811-4f8a-8c1f-3b894975231b","Type":"ContainerDied","Data":"cb26f646072bf84cb21867a8ad46ee883f89c419e810cccbe81f24808d0ccf03"}
Nov 03 20:51:40.158817 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.158719    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb26f646072bf84cb21867a8ad46ee883f89c419e810cccbe81f24808d0ccf03"
Nov 03 20:51:40.333486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.333459    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:40.451444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.451354    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-xks-ca\") pod \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") "
Nov 03 20:51:40.451901 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.451722    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-openshift-ca\") pod \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") "
Nov 03 20:51:40.451901 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.451762    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-kube-api-access-q5kwx\" (UniqueName: \"kubernetes.io/projected/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-kube-api-access-q5kwx\") pod \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\" (UID: \"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909\") "
Nov 03 20:51:40.453010 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.452978    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-openshift-ca" pod "b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909" (UID: "b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:40.454738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.454715    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-kube-api-access-q5kwx" pod "b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909" (UID: "b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909"). InnerVolumeSpecName "kube-api-access-q5kwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:40.455418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.455399    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-xks-ca" pod "b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909" (UID: "b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:40.494602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.494564    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"]
Nov 03 20:51:40.500857 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:40.500820    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207d53e6_6438_445a_a9c5_eda28ffed68f.slice/crio-c6d9a1ade090ea6961544eaa38f6aed0b672545460d6b52cca7b0f34a7c4b8fc WatchSource:0}: Error finding container c6d9a1ade090ea6961544eaa38f6aed0b672545460d6b52cca7b0f34a7c4b8fc: Status 404 returned error can't find the container with id c6d9a1ade090ea6961544eaa38f6aed0b672545460d6b52cca7b0f34a7c4b8fc
Nov 03 20:51:40.503670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.503634    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:40.503747 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.503703    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:40.503794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.503743    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:40.553289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.553261    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:40.553289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.553288    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5kwx\" (UniqueName: \"kubernetes.io/projected/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-kube-api-access-q5kwx\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:40.553479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.553300    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:40.973934 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:40.973899    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:41.170342 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.170303    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" event={"ID":"207d53e6-6438-445a-a9c5-eda28ffed68f","Type":"ContainerStarted","Data":"3e0923e377f8e9d020e347591da9f6fc99e9906aa38ce1275b6edbfb93000a06"}
Nov 03 20:51:41.170520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.170365    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" event={"ID":"207d53e6-6438-445a-a9c5-eda28ffed68f","Type":"ContainerStarted","Data":"c6d9a1ade090ea6961544eaa38f6aed0b672545460d6b52cca7b0f34a7c4b8fc"}
Nov 03 20:51:41.171917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.171888    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"
Nov 03 20:51:41.172030 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.171937    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w" event={"ID":"b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909","Type":"ContainerDied","Data":"048eb4a470e56a101386bccdfa60e8c655b19c3d27319f0943463845fc94a522"}
Nov 03 20:51:41.172030 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.171974    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="048eb4a470e56a101386bccdfa60e8c655b19c3d27319f0943463845fc94a522"
Nov 03 20:51:41.515023 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.514943    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podStartSLOduration=2.5149242320000003 podStartE2EDuration="2.514924232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:41.309842806 +0000 UTC m=+1341.313801530" watchObservedRunningTime="2026-11-03 20:51:41.514924232 +0000 UTC m=+1341.518882958"
Nov 03 20:51:41.520251 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.520224    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"]
Nov 03 20:51:41.521224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.521202    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909" containerName="revoke-keys"
Nov 03 20:51:41.521224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.521223    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909" containerName="revoke-keys"
Nov 03 20:51:41.521403 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.521371    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909" containerName="revoke-keys"
Nov 03 20:51:41.525407 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.525389    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.574684 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.574651    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:51:41.574917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.574869    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:51:41.577256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.577232    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"]
Nov 03 20:51:41.671331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.671297    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.671536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.671363    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49pp\" (UniqueName: \"kubernetes.io/projected/730becb1-edf1-4805-82e2-093baaa9446d-kube-api-access-l49pp\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.671536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.671448    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.772660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.772567    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.772660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.772629    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49pp\" (UniqueName: \"kubernetes.io/projected/730becb1-edf1-4805-82e2-093baaa9446d-kube-api-access-l49pp\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.772855 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.772663    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.773426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.773396    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.775377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.775347    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.827509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.827471    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49pp\" (UniqueName: \"kubernetes.io/projected/730becb1-edf1-4805-82e2-093baaa9446d-kube-api-access-l49pp\") pod \"maas-api-revoke-sub-d2a25245-6kt2x\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:41.866418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:41.866376    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:42.006523 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:42.006490    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:42.008075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:42.008048    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:51:42.008271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:42.008137    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 20:51:42.279501 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:42.279469    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"]
Nov 03 20:51:42.281861 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:42.281827    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730becb1_edf1_4805_82e2_093baaa9446d.slice/crio-3034988881b1c7c532f0532063d64a3adc8a1732b7829c4cd4c29f9f1ec89eb5 WatchSource:0}: Error finding container 3034988881b1c7c532f0532063d64a3adc8a1732b7829c4cd4c29f9f1ec89eb5: Status 404 returned error can't find the container with id 3034988881b1c7c532f0532063d64a3adc8a1732b7829c4cd4c29f9f1ec89eb5
Nov 03 20:51:43.005891 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:43.005851    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:51:43.006411 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:43.005916    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 20:51:43.123996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:43.123959    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:43.192642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:43.192605    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x" event={"ID":"730becb1-edf1-4805-82e2-093baaa9446d","Type":"ContainerStarted","Data":"f7add4df3ae9ebb10da8a9703ea03d6d5850a8776991995812252e46c1e30e29"}
Nov 03 20:51:43.192829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:43.192662    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x" event={"ID":"730becb1-edf1-4805-82e2-093baaa9446d","Type":"ContainerStarted","Data":"3034988881b1c7c532f0532063d64a3adc8a1732b7829c4cd4c29f9f1ec89eb5"}
Nov 03 20:51:43.313777 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:43.313724    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x" podStartSLOduration=2.31370959 podStartE2EDuration="2.31370959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:43.312147386 +0000 UTC m=+1343.316106122" watchObservedRunningTime="2026-11-03 20:51:43.31370959 +0000 UTC m=+1343.317668314"
Nov 03 20:51:44.005786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:44.005753    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:51:44.005944 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:44.005810    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 20:51:44.204576 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:44.204542    5196 generic.go:356] "Generic (PLEG): container finished" podID="730becb1-edf1-4805-82e2-093baaa9446d" containerID="f7add4df3ae9ebb10da8a9703ea03d6d5850a8776991995812252e46c1e30e29" exitCode=0
Nov 03 20:51:44.204765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:44.204627    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x" event={"ID":"730becb1-edf1-4805-82e2-093baaa9446d","Type":"ContainerDied","Data":"f7add4df3ae9ebb10da8a9703ea03d6d5850a8776991995812252e46c1e30e29"}
Nov 03 20:51:45.006412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.006377    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:51:45.006816 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.006438    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 20:51:45.469546 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.469513    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:45.531762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.531728    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-openshift-ca\") pod \"730becb1-edf1-4805-82e2-093baaa9446d\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") "
Nov 03 20:51:45.531966 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.531876    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730becb1-edf1-4805-82e2-093baaa9446d-kube-api-access-l49pp\" (UniqueName: \"kubernetes.io/projected/730becb1-edf1-4805-82e2-093baaa9446d-kube-api-access-l49pp\") pod \"730becb1-edf1-4805-82e2-093baaa9446d\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") "
Nov 03 20:51:45.531966 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.531922    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-xks-ca\") pod \"730becb1-edf1-4805-82e2-093baaa9446d\" (UID: \"730becb1-edf1-4805-82e2-093baaa9446d\") "
Nov 03 20:51:45.532244 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.532209    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-openshift-ca" pod "730becb1-edf1-4805-82e2-093baaa9446d" (UID: "730becb1-edf1-4805-82e2-093baaa9446d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:45.565358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.563831    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-xks-ca" pod "730becb1-edf1-4805-82e2-093baaa9446d" (UID: "730becb1-edf1-4805-82e2-093baaa9446d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:45.565358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.565297    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730becb1-edf1-4805-82e2-093baaa9446d-kube-api-access-l49pp" pod "730becb1-edf1-4805-82e2-093baaa9446d" (UID: "730becb1-edf1-4805-82e2-093baaa9446d"). InnerVolumeSpecName "kube-api-access-l49pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:45.633321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.633274    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:45.633321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.633319    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l49pp\" (UniqueName: \"kubernetes.io/projected/730becb1-edf1-4805-82e2-093baaa9446d-kube-api-access-l49pp\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:45.633321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.633339    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730becb1-edf1-4805-82e2-093baaa9446d-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:45.721496 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:45.721463    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:46.006035 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:46.006000    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:51:46.006217 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:46.006065    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 20:51:46.225461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:46.225433    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"
Nov 03 20:51:46.225461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:46.225451    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x" event={"ID":"730becb1-edf1-4805-82e2-093baaa9446d","Type":"ContainerDied","Data":"3034988881b1c7c532f0532063d64a3adc8a1732b7829c4cd4c29f9f1ec89eb5"}
Nov 03 20:51:46.225933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:46.225503    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3034988881b1c7c532f0532063d64a3adc8a1732b7829c4cd4c29f9f1ec89eb5"
Nov 03 20:51:47.005801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:47.005763    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:51:47.005991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:47.005827    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 20:51:48.011554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:48.011521    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:48.012029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:48.011949    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:48.012665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:48.012647    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 20:51:48.487295 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:51:48.487256    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:51:49.720694 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.720660    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"]
Nov 03 20:51:49.721432 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.721416    5196 cpu_manager.go:414] "containerMap: removing container" podUID="730becb1-edf1-4805-82e2-093baaa9446d" containerName="revoke-keys"
Nov 03 20:51:49.721480 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.721433    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730becb1-edf1-4805-82e2-093baaa9446d" containerName="revoke-keys"
Nov 03 20:51:49.721571 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.721562    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="730becb1-edf1-4805-82e2-093baaa9446d" containerName="revoke-keys"
Nov 03 20:51:49.725748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.725725    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.777340 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.777295    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"]
Nov 03 20:51:49.818266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.818230    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:51:49.818266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.818253    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:51:49.820210 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.820177    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.820471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.820452    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ffj\" (UniqueName: \"kubernetes.io/projected/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-kube-api-access-57ffj\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.820550 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.820494    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.921732 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.921648    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ffj\" (UniqueName: \"kubernetes.io/projected/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-kube-api-access-57ffj\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.921920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.921753    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.921920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.921881    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.922488 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.922461    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.924482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.924461    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:49.977959 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:49.977882    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ffj\" (UniqueName: \"kubernetes.io/projected/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-kube-api-access-57ffj\") pod \"maas-api-revoke-sub-4d8784fa-vf44f\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:50.041195 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:50.041157    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:50.447771 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:50.447745    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"]
Nov 03 20:51:50.450810 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:50.450776    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ad5ec4_ea32_4961_90df_82b5b6ddb920.slice/crio-7aefdcf73e86055aa4c05974dca20f93eaacb190b53dbd4479dea5ba43c0a0ca WatchSource:0}: Error finding container 7aefdcf73e86055aa4c05974dca20f93eaacb190b53dbd4479dea5ba43c0a0ca: Status 404 returned error can't find the container with id 7aefdcf73e86055aa4c05974dca20f93eaacb190b53dbd4479dea5ba43c0a0ca
Nov 03 20:51:50.557240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:50.557206    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:51.048141 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.048080    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"]
Nov 03 20:51:51.094903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.094822    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55802e18-v4lwt"]
Nov 03 20:51:51.104794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.104749    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"]
Nov 03 20:51:51.112707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.112676    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.152272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.152237    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pp4k\" (UniqueName: \"kubernetes.io/projected/e1e280ce-b242-416e-bf3d-e13cd16fa209-kube-api-access-9pp4k\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.152637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.152614    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.152967 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.152947    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.156332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.156304    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"]
Nov 03 20:51:51.254978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.254940    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.255206 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.255025    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp4k\" (UniqueName: \"kubernetes.io/projected/e1e280ce-b242-416e-bf3d-e13cd16fa209-kube-api-access-9pp4k\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.255206 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.255159    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.255702 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.255680    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.257765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.257742    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.273182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.273150    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f" event={"ID":"a6ad5ec4-ea32-4961-90df-82b5b6ddb920","Type":"ContainerStarted","Data":"3f583dcc770a099917b3c38a50e6d2575d92c9d36e95d664736d4d120af3cf93"}
Nov 03 20:51:51.273346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.273204    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f" event={"ID":"a6ad5ec4-ea32-4961-90df-82b5b6ddb920","Type":"ContainerStarted","Data":"7aefdcf73e86055aa4c05974dca20f93eaacb190b53dbd4479dea5ba43c0a0ca"}
Nov 03 20:51:51.311009 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.310970    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp4k\" (UniqueName: \"kubernetes.io/projected/e1e280ce-b242-416e-bf3d-e13cd16fa209-kube-api-access-9pp4k\") pod \"maas-api-revoke-sub-9f054dfe-tq4zp\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.389388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.389261    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f" podStartSLOduration=2.389246401 podStartE2EDuration="2.389246401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:51.387467946 +0000 UTC m=+1351.391426679" watchObservedRunningTime="2026-11-03 20:51:51.389246401 +0000 UTC m=+1351.393205125"
Nov 03 20:51:51.429351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.429311    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:51.814467 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:51.814436    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"]
Nov 03 20:51:51.816690 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:51.816657    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e280ce_b242_416e_bf3d_e13cd16fa209.slice/crio-5af833973c9253eefa4841562a0d44810b09a64f475914ba6de97916eafab3d0 WatchSource:0}: Error finding container 5af833973c9253eefa4841562a0d44810b09a64f475914ba6de97916eafab3d0: Status 404 returned error can't find the container with id 5af833973c9253eefa4841562a0d44810b09a64f475914ba6de97916eafab3d0
Nov 03 20:51:52.111454 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.111418    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e454dd07-e27d-4498-ac05-53ccf1f341b6" path="/var/lib/kubelet/pods/e454dd07-e27d-4498-ac05-53ccf1f341b6/volumes"
Nov 03 20:51:52.224922 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.224882    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"]
Nov 03 20:51:52.231155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.231131    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.270269 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.270238    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.270406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.270289    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprk4\" (UniqueName: \"kubernetes.io/projected/462cb87f-6a9b-47fa-87c5-8f783a7650fb-kube-api-access-jprk4\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.270406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.270378    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.282541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.282506    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"]
Nov 03 20:51:52.288548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.288503    5196 generic.go:356] "Generic (PLEG): container finished" podID="a6ad5ec4-ea32-4961-90df-82b5b6ddb920" containerID="3f583dcc770a099917b3c38a50e6d2575d92c9d36e95d664736d4d120af3cf93" exitCode=0
Nov 03 20:51:52.288738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.288617    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f" event={"ID":"a6ad5ec4-ea32-4961-90df-82b5b6ddb920","Type":"ContainerDied","Data":"3f583dcc770a099917b3c38a50e6d2575d92c9d36e95d664736d4d120af3cf93"}
Nov 03 20:51:52.291052 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.290969    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp" event={"ID":"e1e280ce-b242-416e-bf3d-e13cd16fa209","Type":"ContainerStarted","Data":"d960c5dacb4b91ed21d633fd244a846af04974fa687ff2d8ad4b834a79d9baeb"}
Nov 03 20:51:52.291052 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.291048    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp" event={"ID":"e1e280ce-b242-416e-bf3d-e13cd16fa209","Type":"ContainerStarted","Data":"5af833973c9253eefa4841562a0d44810b09a64f475914ba6de97916eafab3d0"}
Nov 03 20:51:52.372116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.371997    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprk4\" (UniqueName: \"kubernetes.io/projected/462cb87f-6a9b-47fa-87c5-8f783a7650fb-kube-api-access-jprk4\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.372116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.372068    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.372480 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.372456    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.372848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.372827    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.375132 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.375107    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.426317 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.426285    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprk4\" (UniqueName: \"kubernetes.io/projected/462cb87f-6a9b-47fa-87c5-8f783a7650fb-kube-api-access-jprk4\") pod \"maas-api-revoke-sub-82a67aed-2d7sm\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.547464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.547427    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:52.551935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.551878    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp" podStartSLOduration=1.551860408 podStartE2EDuration="1.551860408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:52.551602331 +0000 UTC m=+1352.555561056" watchObservedRunningTime="2026-11-03 20:51:52.551860408 +0000 UTC m=+1352.555819135"
Nov 03 20:51:52.927799 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.927770    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"]
Nov 03 20:51:52.959332 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:52.959296    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462cb87f_6a9b_47fa_87c5_8f783a7650fb.slice/crio-1d5923081f1cdff9f6215346f2a707b5f83e9ae00123c9bfd01b25ac3f68415a WatchSource:0}: Error finding container 1d5923081f1cdff9f6215346f2a707b5f83e9ae00123c9bfd01b25ac3f68415a: Status 404 returned error can't find the container with id 1d5923081f1cdff9f6215346f2a707b5f83e9ae00123c9bfd01b25ac3f68415a
Nov 03 20:51:52.984983 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.984950    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"]
Nov 03 20:51:52.991020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:52.990998    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.043243 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.043212    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"]
Nov 03 20:51:53.084399 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.084359    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fe2825-openshift-default-dockercfg-7zsdg\""
Nov 03 20:51:53.084576 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.084521    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdf8d45-c633-456c-a6f0-268812229551-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.084717 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.084694    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdf8d45-c633-456c-a6f0-268812229551-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.084785 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.084739    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.084835 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.084787    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.084892 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.084876    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.084965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.084918    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.085030 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.084964    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ph7f\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-kube-api-access-6ph7f\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.085030 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.085003    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.085145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.085072    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.186611 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.186562    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.186624    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.186688    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ph7f\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-kube-api-access-6ph7f\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.186745    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.186826    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.186982    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdf8d45-c633-456c-a6f0-268812229551-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.187120    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.187144    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdf8d45-c633-456c-a6f0-268812229551-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.187172    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.187187    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.187237    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.187341    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.187605    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.187871 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.187849    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdf8d45-c633-456c-a6f0-268812229551-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.189852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.189821    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.189964 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.189941    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdf8d45-c633-456c-a6f0-268812229551-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.242897 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.242860    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.243086 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.242905    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ph7f\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-kube-api-access-6ph7f\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.302618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.302582    5196 generic.go:356] "Generic (PLEG): container finished" podID="e1e280ce-b242-416e-bf3d-e13cd16fa209" containerID="d960c5dacb4b91ed21d633fd244a846af04974fa687ff2d8ad4b834a79d9baeb" exitCode=0
Nov 03 20:51:53.302775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.302676    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp" event={"ID":"e1e280ce-b242-416e-bf3d-e13cd16fa209","Type":"ContainerDied","Data":"d960c5dacb4b91ed21d633fd244a846af04974fa687ff2d8ad4b834a79d9baeb"}
Nov 03 20:51:53.304309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.304283    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm" event={"ID":"462cb87f-6a9b-47fa-87c5-8f783a7650fb","Type":"ContainerStarted","Data":"1d5923081f1cdff9f6215346f2a707b5f83e9ae00123c9bfd01b25ac3f68415a"}
Nov 03 20:51:53.327395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.327360    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:53.487227 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.487197    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:53.595064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.594911    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-xks-ca\") pod \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") "
Nov 03 20:51:53.595280 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.595080    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-openshift-ca\") pod \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") "
Nov 03 20:51:53.595280 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.595272    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-kube-api-access-57ffj\" (UniqueName: \"kubernetes.io/projected/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-kube-api-access-57ffj\") pod \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\" (UID: \"a6ad5ec4-ea32-4961-90df-82b5b6ddb920\") "
Nov 03 20:51:53.596569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.595637    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-openshift-ca" pod "a6ad5ec4-ea32-4961-90df-82b5b6ddb920" (UID: "a6ad5ec4-ea32-4961-90df-82b5b6ddb920"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:53.596569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.595932    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:53.598393 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.598333    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-xks-ca" pod "a6ad5ec4-ea32-4961-90df-82b5b6ddb920" (UID: "a6ad5ec4-ea32-4961-90df-82b5b6ddb920"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:53.598722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.598684    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-kube-api-access-57ffj" pod "a6ad5ec4-ea32-4961-90df-82b5b6ddb920" (UID: "a6ad5ec4-ea32-4961-90df-82b5b6ddb920"). InnerVolumeSpecName "kube-api-access-57ffj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:53.697064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.696973    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57ffj\" (UniqueName: \"kubernetes.io/projected/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-kube-api-access-57ffj\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:53.697064 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.697008    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ad5ec4-ea32-4961-90df-82b5b6ddb920-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:53.810180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.810148    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"]
Nov 03 20:51:53.813136 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:53.813085    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdf8d45_c633_456c_a6f0_268812229551.slice/crio-2e021416b8f5f0a7f1f4abd7ec478b165831a54d6845fcfe7c132ca6ecd9d49b WatchSource:0}: Error finding container 2e021416b8f5f0a7f1f4abd7ec478b165831a54d6845fcfe7c132ca6ecd9d49b: Status 404 returned error can't find the container with id 2e021416b8f5f0a7f1f4abd7ec478b165831a54d6845fcfe7c132ca6ecd9d49b
Nov 03 20:51:53.815597 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.815561    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:53.815686 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.815654    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:53.815739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:53.815687    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:54.049207 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.047957    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"]
Nov 03 20:51:54.053665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.053634    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceebcd8-zbtb9"]
Nov 03 20:51:54.112556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.112520    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859533d1-0dc9-40dc-8baf-d494518ba039" path="/var/lib/kubelet/pods/859533d1-0dc9-40dc-8baf-d494518ba039/volumes"
Nov 03 20:51:54.315563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.315470    5196 generic.go:356] "Generic (PLEG): container finished" podID="462cb87f-6a9b-47fa-87c5-8f783a7650fb" containerID="a45e163ee6b7707f9d75fac9266dd91ed659d9bbb2a5a9c658c2d948c9295138" exitCode=0
Nov 03 20:51:54.315973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.315560    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm" event={"ID":"462cb87f-6a9b-47fa-87c5-8f783a7650fb","Type":"ContainerDied","Data":"a45e163ee6b7707f9d75fac9266dd91ed659d9bbb2a5a9c658c2d948c9295138"}
Nov 03 20:51:54.317888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.317856    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" event={"ID":"3bdf8d45-c633-456c-a6f0-268812229551","Type":"ContainerStarted","Data":"cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52"}
Nov 03 20:51:54.318080 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.317923    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" event={"ID":"3bdf8d45-c633-456c-a6f0-268812229551","Type":"ContainerStarted","Data":"2e021416b8f5f0a7f1f4abd7ec478b165831a54d6845fcfe7c132ca6ecd9d49b"}
Nov 03 20:51:54.319905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.319883    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"
Nov 03 20:51:54.320034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.319913    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f" event={"ID":"a6ad5ec4-ea32-4961-90df-82b5b6ddb920","Type":"ContainerDied","Data":"7aefdcf73e86055aa4c05974dca20f93eaacb190b53dbd4479dea5ba43c0a0ca"}
Nov 03 20:51:54.320034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.319957    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aefdcf73e86055aa4c05974dca20f93eaacb190b53dbd4479dea5ba43c0a0ca"
Nov 03 20:51:54.328170 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.328136    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:51:54.330544 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.330503    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 20:51:54.330717 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.330571    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 20:51:54.574750 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.574724    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:54.618797 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.618765    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-openshift-ca\") pod \"e1e280ce-b242-416e-bf3d-e13cd16fa209\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") "
Nov 03 20:51:54.618988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.618975    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-xks-ca\") pod \"e1e280ce-b242-416e-bf3d-e13cd16fa209\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") "
Nov 03 20:51:54.619037 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.619022    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e280ce-b242-416e-bf3d-e13cd16fa209-kube-api-access-9pp4k\" (UniqueName: \"kubernetes.io/projected/e1e280ce-b242-416e-bf3d-e13cd16fa209-kube-api-access-9pp4k\") pod \"e1e280ce-b242-416e-bf3d-e13cd16fa209\" (UID: \"e1e280ce-b242-416e-bf3d-e13cd16fa209\") "
Nov 03 20:51:54.619231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.619201    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-openshift-ca" pod "e1e280ce-b242-416e-bf3d-e13cd16fa209" (UID: "e1e280ce-b242-416e-bf3d-e13cd16fa209"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:54.619540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.619512    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:54.621323 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.621300    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-xks-ca" pod "e1e280ce-b242-416e-bf3d-e13cd16fa209" (UID: "e1e280ce-b242-416e-bf3d-e13cd16fa209"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:54.621500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.621477    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e280ce-b242-416e-bf3d-e13cd16fa209-kube-api-access-9pp4k" pod "e1e280ce-b242-416e-bf3d-e13cd16fa209" (UID: "e1e280ce-b242-416e-bf3d-e13cd16fa209"). InnerVolumeSpecName "kube-api-access-9pp4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:54.720642 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.720594    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pp4k\" (UniqueName: \"kubernetes.io/projected/e1e280ce-b242-416e-bf3d-e13cd16fa209-kube-api-access-9pp4k\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:54.720888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.720867    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e280ce-b242-416e-bf3d-e13cd16fa209-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:54.727670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.727605    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podStartSLOduration=2.727583862 podStartE2EDuration="2.727583862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:54.565552615 +0000 UTC m=+1354.569511360" watchObservedRunningTime="2026-11-03 20:51:54.727583862 +0000 UTC m=+1354.731542587"
Nov 03 20:51:54.783791 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.783754    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"]
Nov 03 20:51:54.784882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.784830    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a6ad5ec4-ea32-4961-90df-82b5b6ddb920" containerName="revoke-keys"
Nov 03 20:51:54.784882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.784860    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ad5ec4-ea32-4961-90df-82b5b6ddb920" containerName="revoke-keys"
Nov 03 20:51:54.784882 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.784886    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e1e280ce-b242-416e-bf3d-e13cd16fa209" containerName="revoke-keys"
Nov 03 20:51:54.785147 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.784893    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e280ce-b242-416e-bf3d-e13cd16fa209" containerName="revoke-keys"
Nov 03 20:51:54.785228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.785200    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ad5ec4-ea32-4961-90df-82b5b6ddb920" containerName="revoke-keys"
Nov 03 20:51:54.785228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.785221    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e280ce-b242-416e-bf3d-e13cd16fa209" containerName="revoke-keys"
Nov 03 20:51:54.789775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.789755    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.882598 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.882511    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"]
Nov 03 20:51:54.924538 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924501    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.924710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924540    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.924710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924651    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.924710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924710    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.924832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924776    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.924832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924797    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.924908 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924844    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.924908 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924861    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:54.924978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:54.924907    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxs2h\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-kube-api-access-dxs2h\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026194 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026128    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026194 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026195    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026262    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026284    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026352    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxs2h\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-kube-api-access-dxs2h\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026397    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026422    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026559    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026647    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026705    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026753    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.026893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.026869    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.027179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.027152    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.027398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.027345    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.028888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.028862    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.028966 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.028948    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.080606 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.080578    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.080759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.080642    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxs2h\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-kube-api-access-dxs2h\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.106721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.106684    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:55.328540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.328494    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 20:51:55.329022 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.328563    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 20:51:55.331548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.331523    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"
Nov 03 20:51:55.331548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.331529    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp" event={"ID":"e1e280ce-b242-416e-bf3d-e13cd16fa209","Type":"ContainerDied","Data":"5af833973c9253eefa4841562a0d44810b09a64f475914ba6de97916eafab3d0"}
Nov 03 20:51:55.331781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.331577    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af833973c9253eefa4841562a0d44810b09a64f475914ba6de97916eafab3d0"
Nov 03 20:51:55.475406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.473216    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:55.616237 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.616211    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:55.628355 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.628321    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"]
Nov 03 20:51:55.630911 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:55.630883    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c3a3a5_1657_4d9f_875e_ac5e00c808a6.slice/crio-74751759c38f25bb17a32ec984186f2561fbc7df2930af207cdd7f5fb92746b3 WatchSource:0}: Error finding container 74751759c38f25bb17a32ec984186f2561fbc7df2930af207cdd7f5fb92746b3: Status 404 returned error can't find the container with id 74751759c38f25bb17a32ec984186f2561fbc7df2930af207cdd7f5fb92746b3
Nov 03 20:51:55.633864 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.633827    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:55.633964 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.633947    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:55.634020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.633998    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:55.768700 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.768625    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462cb87f-6a9b-47fa-87c5-8f783a7650fb-kube-api-access-jprk4\" (UniqueName: \"kubernetes.io/projected/462cb87f-6a9b-47fa-87c5-8f783a7650fb-kube-api-access-jprk4\") pod \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") "
Nov 03 20:51:55.769117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.769052    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-openshift-ca\") pod \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") "
Nov 03 20:51:55.770439 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.770387    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-openshift-ca" pod "462cb87f-6a9b-47fa-87c5-8f783a7650fb" (UID: "462cb87f-6a9b-47fa-87c5-8f783a7650fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:55.771034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.770908    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-xks-ca\") pod \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\" (UID: \"462cb87f-6a9b-47fa-87c5-8f783a7650fb\") "
Nov 03 20:51:55.772023 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.772001    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:55.779342 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.779310    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462cb87f-6a9b-47fa-87c5-8f783a7650fb-kube-api-access-jprk4" pod "462cb87f-6a9b-47fa-87c5-8f783a7650fb" (UID: "462cb87f-6a9b-47fa-87c5-8f783a7650fb"). InnerVolumeSpecName "kube-api-access-jprk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:55.780226 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.780201    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-xks-ca" pod "462cb87f-6a9b-47fa-87c5-8f783a7650fb" (UID: "462cb87f-6a9b-47fa-87c5-8f783a7650fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:55.875143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.873817    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cb87f-6a9b-47fa-87c5-8f783a7650fb-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:55.875143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.873853    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jprk4\" (UniqueName: \"kubernetes.io/projected/462cb87f-6a9b-47fa-87c5-8f783a7650fb-kube-api-access-jprk4\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:55.929202 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.929164    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"]
Nov 03 20:51:55.929989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.929965    5196 cpu_manager.go:414] "containerMap: removing container" podUID="462cb87f-6a9b-47fa-87c5-8f783a7650fb" containerName="revoke-keys"
Nov 03 20:51:55.930115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.929990    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462cb87f-6a9b-47fa-87c5-8f783a7650fb" containerName="revoke-keys"
Nov 03 20:51:55.930257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.930241    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="462cb87f-6a9b-47fa-87c5-8f783a7650fb" containerName="revoke-keys"
Nov 03 20:51:55.962116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.960415    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:55.989276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:55.989222    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"]
Nov 03 20:51:56.076627 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.076595    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhbs\" (UniqueName: \"kubernetes.io/projected/1dc601ea-c169-4823-93a1-0815589780dd-kube-api-access-5dhbs\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.076781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.076720    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.076781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.076765    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.178845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.178799    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.179016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.178890    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.179157 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.179141    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhbs\" (UniqueName: \"kubernetes.io/projected/1dc601ea-c169-4823-93a1-0815589780dd-kube-api-access-5dhbs\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.179557 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.179533    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.181787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.181762    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.233418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.233385    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhbs\" (UniqueName: \"kubernetes.io/projected/1dc601ea-c169-4823-93a1-0815589780dd-kube-api-access-5dhbs\") pod \"maas-api-revoke-sub-646f8fbe-qjb8t\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.300504 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.300401    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:56.328128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.328078    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 20:51:56.328326 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.328157    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 20:51:56.373541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.373506    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"
Nov 03 20:51:56.374741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.373621    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm" event={"ID":"462cb87f-6a9b-47fa-87c5-8f783a7650fb","Type":"ContainerDied","Data":"1d5923081f1cdff9f6215346f2a707b5f83e9ae00123c9bfd01b25ac3f68415a"}
Nov 03 20:51:56.374741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.373678    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5923081f1cdff9f6215346f2a707b5f83e9ae00123c9bfd01b25ac3f68415a"
Nov 03 20:51:56.378849 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.378597    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" event={"ID":"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6","Type":"ContainerStarted","Data":"f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442"}
Nov 03 20:51:56.378849 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.378657    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" event={"ID":"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6","Type":"ContainerStarted","Data":"74751759c38f25bb17a32ec984186f2561fbc7df2930af207cdd7f5fb92746b3"}
Nov 03 20:51:56.494136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.494056    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podStartSLOduration=2.494037371 podStartE2EDuration="2.494037371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:56.488729027 +0000 UTC m=+1356.492687754" watchObservedRunningTime="2026-11-03 20:51:56.494037371 +0000 UTC m=+1356.497996095"
Nov 03 20:51:56.731528 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:51:56.731469    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc601ea_c169_4823_93a1_0815589780dd.slice/crio-d735b5ecd514860f483ac113e71134a9eee626d247048b71b1540ff846a9573d WatchSource:0}: Error finding container d735b5ecd514860f483ac113e71134a9eee626d247048b71b1540ff846a9573d: Status 404 returned error can't find the container with id d735b5ecd514860f483ac113e71134a9eee626d247048b71b1540ff846a9573d
Nov 03 20:51:56.747139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:56.734216    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"]
Nov 03 20:51:57.001368 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:51:56.999522    5196 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc601ea_c169_4823_93a1_0815589780dd.slice/crio-1ffa5b1342bce36491aa3e9dc78469d3ad2c0be933610e28ac794a0b0dec0b03.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:51:57.107182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:57.107143    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:51:57.108775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:57.108744    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 20:51:57.108926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:57.108797    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 20:51:57.328463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:57.328429    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 20:51:57.328686 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:57.328486    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 20:51:57.399288 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:57.399253    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t" event={"ID":"1dc601ea-c169-4823-93a1-0815589780dd","Type":"ContainerStarted","Data":"1ffa5b1342bce36491aa3e9dc78469d3ad2c0be933610e28ac794a0b0dec0b03"}
Nov 03 20:51:57.399733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:57.399714    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t" event={"ID":"1dc601ea-c169-4823-93a1-0815589780dd","Type":"ContainerStarted","Data":"d735b5ecd514860f483ac113e71134a9eee626d247048b71b1540ff846a9573d"}
Nov 03 20:51:57.528608 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:57.521243    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t" podStartSLOduration=2.521221003 podStartE2EDuration="2.521221003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:57.510158808 +0000 UTC m=+1357.514117532" watchObservedRunningTime="2026-11-03 20:51:57.521221003 +0000 UTC m=+1357.525179731"
Nov 03 20:51:58.086122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.085813    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"]
Nov 03 20:51:58.097020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.096971    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e0d8c9-ls7d8"]
Nov 03 20:51:58.107753 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.107722    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 20:51:58.107954 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.107778    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 20:51:58.111330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.111298    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74351c3a-db06-4887-9edb-8edd08a43b37" path="/var/lib/kubelet/pods/74351c3a-db06-4887-9edb-8edd08a43b37/volumes"
Nov 03 20:51:58.328021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.327991    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 20:51:58.328208 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.328050    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 20:51:58.410874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.410838    5196 generic.go:356] "Generic (PLEG): container finished" podID="1dc601ea-c169-4823-93a1-0815589780dd" containerID="1ffa5b1342bce36491aa3e9dc78469d3ad2c0be933610e28ac794a0b0dec0b03" exitCode=0
Nov 03 20:51:58.411447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:58.410911    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t" event={"ID":"1dc601ea-c169-4823-93a1-0815589780dd","Type":"ContainerDied","Data":"1ffa5b1342bce36491aa3e9dc78469d3ad2c0be933610e28ac794a0b0dec0b03"}
Nov 03 20:51:59.107998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.107962    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 20:51:59.108179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.108014    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 20:51:59.328051 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.328007    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 20:51:59.328260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.328071    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 20:51:59.806764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.806723    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:51:59.972178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.972040    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-openshift-ca\") pod \"1dc601ea-c169-4823-93a1-0815589780dd\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") "
Nov 03 20:51:59.972178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.972117    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc601ea-c169-4823-93a1-0815589780dd-kube-api-access-5dhbs\" (UniqueName: \"kubernetes.io/projected/1dc601ea-c169-4823-93a1-0815589780dd-kube-api-access-5dhbs\") pod \"1dc601ea-c169-4823-93a1-0815589780dd\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") "
Nov 03 20:51:59.972420 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.972187    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-xks-ca\") pod \"1dc601ea-c169-4823-93a1-0815589780dd\" (UID: \"1dc601ea-c169-4823-93a1-0815589780dd\") "
Nov 03 20:51:59.978344 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.978301    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-openshift-ca" pod "1dc601ea-c169-4823-93a1-0815589780dd" (UID: "1dc601ea-c169-4823-93a1-0815589780dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:59.980475 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.980422    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-xks-ca" pod "1dc601ea-c169-4823-93a1-0815589780dd" (UID: "1dc601ea-c169-4823-93a1-0815589780dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:59.980848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:51:59.980817    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc601ea-c169-4823-93a1-0815589780dd-kube-api-access-5dhbs" pod "1dc601ea-c169-4823-93a1-0815589780dd" (UID: "1dc601ea-c169-4823-93a1-0815589780dd"). InnerVolumeSpecName "kube-api-access-5dhbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:00.076685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.076628    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:00.078144 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.078119    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dhbs\" (UniqueName: \"kubernetes.io/projected/1dc601ea-c169-4823-93a1-0815589780dd-kube-api-access-5dhbs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:00.078312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.078295    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc601ea-c169-4823-93a1-0815589780dd-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:00.109897 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.109859    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 20:52:00.110123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.109913    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 20:52:00.327948 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.327913    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 20:52:00.328179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.327975    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 20:52:00.439118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.436268    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t" event={"ID":"1dc601ea-c169-4823-93a1-0815589780dd","Type":"ContainerDied","Data":"d735b5ecd514860f483ac113e71134a9eee626d247048b71b1540ff846a9573d"}
Nov 03 20:52:00.439118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.436336    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d735b5ecd514860f483ac113e71134a9eee626d247048b71b1540ff846a9573d"
Nov 03 20:52:00.439118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.436409    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"
Nov 03 20:52:00.507478 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.507442    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:00.767181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:00.758315    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:01.107412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.107380    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 20:52:01.107787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.107435    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 20:52:01.581738 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.581700    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:52:01.582763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.582738    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:52:01.583669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.583652    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:52:01.865904 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.865804    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"]
Nov 03 20:52:01.866927 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.866904    5196 cpu_manager.go:414] "containerMap: removing container" podUID="1dc601ea-c169-4823-93a1-0815589780dd" containerName="revoke-keys"
Nov 03 20:52:01.867001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.866930    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc601ea-c169-4823-93a1-0815589780dd" containerName="revoke-keys"
Nov 03 20:52:01.867124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.867112    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc601ea-c169-4823-93a1-0815589780dd" containerName="revoke-keys"
Nov 03 20:52:01.871268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.871247    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:01.920551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.920514    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:52:01.920745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.920598    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:52:01.925503 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:01.925468    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"]
Nov 03 20:52:02.023521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.023482    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.023679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.023530    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.023741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.023723    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8j29\" (UniqueName: \"kubernetes.io/projected/6f7818f4-eb56-4992-abde-a0fa52f81869-kube-api-access-p8j29\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.107585 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.107545    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 20:52:02.108031 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.107623    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 20:52:02.124960 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.124875    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8j29\" (UniqueName: \"kubernetes.io/projected/6f7818f4-eb56-4992-abde-a0fa52f81869-kube-api-access-p8j29\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.125179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.124999    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.125179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.125023    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.125680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.125658    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.127690 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.127665    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.179574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.179541    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8j29\" (UniqueName: \"kubernetes.io/projected/6f7818f4-eb56-4992-abde-a0fa52f81869-kube-api-access-p8j29\") pod \"maas-api-revoke-sub-3b3d7910-5lns9\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.189573 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.189546    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:02.599077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:02.599051    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"]
Nov 03 20:52:02.603354 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:02.603323    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7818f4_eb56_4992_abde_a0fa52f81869.slice/crio-eea620f7947ed799c2cd8d3fe8aed5985a730e5bfef4d058955acb51fe53b870 WatchSource:0}: Error finding container eea620f7947ed799c2cd8d3fe8aed5985a730e5bfef4d058955acb51fe53b870: Status 404 returned error can't find the container with id eea620f7947ed799c2cd8d3fe8aed5985a730e5bfef4d058955acb51fe53b870
Nov 03 20:52:03.107888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:03.107790    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 03 20:52:03.107888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:03.107864    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 03 20:52:03.496898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:03.496799    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9" event={"ID":"6f7818f4-eb56-4992-abde-a0fa52f81869","Type":"ContainerStarted","Data":"bb5f5d8d937d82a19533fe901b324692f918344f432d21764072bd5512a2f7e5"}
Nov 03 20:52:03.496898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:03.496856    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9" event={"ID":"6f7818f4-eb56-4992-abde-a0fa52f81869","Type":"ContainerStarted","Data":"eea620f7947ed799c2cd8d3fe8aed5985a730e5bfef4d058955acb51fe53b870"}
Nov 03 20:52:03.605553 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:03.605487    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9" podStartSLOduration=2.6054676839999997 podStartE2EDuration="2.605467684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:03.602785319 +0000 UTC m=+1363.606744069" watchObservedRunningTime="2026-11-03 20:52:03.605467684 +0000 UTC m=+1363.609426408"
Nov 03 20:52:04.112768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:04.112738    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:52:04.113236 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:04.113219    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:52:04.113923 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:04.113904    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 20:52:04.506635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:04.506549    5196 generic.go:356] "Generic (PLEG): container finished" podID="6f7818f4-eb56-4992-abde-a0fa52f81869" containerID="bb5f5d8d937d82a19533fe901b324692f918344f432d21764072bd5512a2f7e5" exitCode=0
Nov 03 20:52:04.506777 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:04.506629    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9" event={"ID":"6f7818f4-eb56-4992-abde-a0fa52f81869","Type":"ContainerDied","Data":"bb5f5d8d937d82a19533fe901b324692f918344f432d21764072bd5512a2f7e5"}
Nov 03 20:52:05.265981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:05.265944    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:05.890733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:05.890680    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:06.007460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:05.970084    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-xks-ca\") pod \"6f7818f4-eb56-4992-abde-a0fa52f81869\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") "
Nov 03 20:52:06.007460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:05.970195    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7818f4-eb56-4992-abde-a0fa52f81869-kube-api-access-p8j29\" (UniqueName: \"kubernetes.io/projected/6f7818f4-eb56-4992-abde-a0fa52f81869-kube-api-access-p8j29\") pod \"6f7818f4-eb56-4992-abde-a0fa52f81869\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") "
Nov 03 20:52:06.007460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:05.970352    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-openshift-ca\") pod \"6f7818f4-eb56-4992-abde-a0fa52f81869\" (UID: \"6f7818f4-eb56-4992-abde-a0fa52f81869\") "
Nov 03 20:52:06.010343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.010207    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-xks-ca" pod "6f7818f4-eb56-4992-abde-a0fa52f81869" (UID: "6f7818f4-eb56-4992-abde-a0fa52f81869"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:06.010745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.010699    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-openshift-ca" pod "6f7818f4-eb56-4992-abde-a0fa52f81869" (UID: "6f7818f4-eb56-4992-abde-a0fa52f81869"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:06.038197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.038046    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7818f4-eb56-4992-abde-a0fa52f81869-kube-api-access-p8j29" pod "6f7818f4-eb56-4992-abde-a0fa52f81869" (UID: "6f7818f4-eb56-4992-abde-a0fa52f81869"). InnerVolumeSpecName "kube-api-access-p8j29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:06.091522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.078673    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:06.091522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.078716    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7818f4-eb56-4992-abde-a0fa52f81869-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:06.091522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.078732    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8j29\" (UniqueName: \"kubernetes.io/projected/6f7818f4-eb56-4992-abde-a0fa52f81869-kube-api-access-p8j29\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:06.604012 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.603967    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9" event={"ID":"6f7818f4-eb56-4992-abde-a0fa52f81869","Type":"ContainerDied","Data":"eea620f7947ed799c2cd8d3fe8aed5985a730e5bfef4d058955acb51fe53b870"}
Nov 03 20:52:06.604649 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.604624    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eea620f7947ed799c2cd8d3fe8aed5985a730e5bfef4d058955acb51fe53b870"
Nov 03 20:52:06.604837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:06.604823    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"
Nov 03 20:52:07.376306 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:07.376228    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:07.584982 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:07.584257    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:07.680324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:07.680217    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:07.697715 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:07.697670    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:07.997181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:07.995212    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 20:52:08.056174 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.051448    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:08.468118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.443545    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:52:08.494119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.468919    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:52:08.680129 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.529300    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 03 20:52:08.712710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.587348    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": context deadline exceeded"
Nov 03 20:52:08.712710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.587387    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/health\": context deadline exceeded"
Nov 03 20:52:08.712710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.587507    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:08.712710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.659806    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": context deadline exceeded"
Nov 03 20:52:08.712710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.699814    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:52:08.712710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.710730    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:08.712710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.711074    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:52:08.712710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.712035    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:08.777224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.762663    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:08.777224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:08.762728    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:09.098225 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.097964    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": context deadline exceeded"
Nov 03 20:52:09.099840 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.099814    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:52:09.132943 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.132895    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 20:52:09.142538 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.142483    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:52:09.729184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.729137    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:09.796574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.729789    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 03 20:52:09.796888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.738751    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:09.797045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.738877    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:09.797229 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.738986    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:09.797586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.739083    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": context deadline exceeded"
Nov 03 20:52:09.805992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.739202    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 20:52:09.806272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.739302    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:09.806394 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.771685    5196 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-v455q container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.22:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:09.890129 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.886632    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-v455q" podUID="170428c7-02e4-40f5-a88c-1634d8b58b5a" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.22:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:09.890811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.890787    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:52:09.929163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.929113    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:52:09.929400 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.929188    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:52:09.929400 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:09.929246    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:52:10.041483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.041396    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:52:10.041810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.041780    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:52:10.042032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.042020    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:52:10.043631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.043612    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:52:10.043838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.043814    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:52:10.062143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.061628    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:52:10.129276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.129245    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"]
Nov 03 20:52:10.129564 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.129546    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:52:10.129730 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.129717    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:52:10.129904 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.129890    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:52:10.131653 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.131634    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"]
Nov 03 20:52:10.141725 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.141699    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae938b92-7469f"]
Nov 03 20:52:10.150653 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.150628    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1f420c-dpg8l"]
Nov 03 20:52:10.663607 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.663573    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"]
Nov 03 20:52:10.664377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.664360    5196 cpu_manager.go:414] "containerMap: removing container" podUID="6f7818f4-eb56-4992-abde-a0fa52f81869" containerName="revoke-keys"
Nov 03 20:52:10.664377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.664377    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7818f4-eb56-4992-abde-a0fa52f81869" containerName="revoke-keys"
Nov 03 20:52:10.664509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.664483    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7818f4-eb56-4992-abde-a0fa52f81869" containerName="revoke-keys"
Nov 03 20:52:10.669430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.669409    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.720857 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.720828    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:52:10.721048 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.720997    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:52:10.723975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.723947    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"]
Nov 03 20:52:10.759309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.759274    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.759798 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.759412    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.759798 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.759479    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9wj\" (UniqueName: \"kubernetes.io/projected/86a76380-07a5-487e-a82e-23dd52b7cfd5-kube-api-access-9w9wj\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.861419 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.861380    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.861606 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.861446    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.861606 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.861472    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9wj\" (UniqueName: \"kubernetes.io/projected/86a76380-07a5-487e-a82e-23dd52b7cfd5-kube-api-access-9w9wj\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.862209 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.862184    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.864221 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.864196    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.918958 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.918882    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9wj\" (UniqueName: \"kubernetes.io/projected/86a76380-07a5-487e-a82e-23dd52b7cfd5-kube-api-access-9w9wj\") pod \"maas-api-revoke-sub-d8179659-9fp2z\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:10.984672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:10.984637    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:11.110271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:11.110231    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:11.401622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:11.401573    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"]
Nov 03 20:52:11.405983 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:11.405944    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a76380_07a5_487e_a82e_23dd52b7cfd5.slice/crio-c4ef055330006c838cbdbfbfb7f42d6c11817d53bf69f024c66848bdf2e28c42 WatchSource:0}: Error finding container c4ef055330006c838cbdbfbfb7f42d6c11817d53bf69f024c66848bdf2e28c42: Status 404 returned error can't find the container with id c4ef055330006c838cbdbfbfb7f42d6c11817d53bf69f024c66848bdf2e28c42
Nov 03 20:52:12.009392 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.009292    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"]
Nov 03 20:52:12.014768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.014742    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.069434 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.069363    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"]
Nov 03 20:52:12.090419 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.090381    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.090682 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.090661    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.090821 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.090802    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8g2m\" (UniqueName: \"kubernetes.io/projected/a22f1f1a-d59b-476d-8474-d6d611be0d64-kube-api-access-m8g2m\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.092875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.092833    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z" event={"ID":"86a76380-07a5-487e-a82e-23dd52b7cfd5","Type":"ContainerStarted","Data":"4962933513b7cd43d2a22ff8d411c0b79ad1f80d0a5285a5496a7d378010f80d"}
Nov 03 20:52:12.093051 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.092906    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z" event={"ID":"86a76380-07a5-487e-a82e-23dd52b7cfd5","Type":"ContainerStarted","Data":"c4ef055330006c838cbdbfbfb7f42d6c11817d53bf69f024c66848bdf2e28c42"}
Nov 03 20:52:12.111611 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.111579    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb922f2e-feb3-4249-a4a4-052ae2f4c6a0" path="/var/lib/kubelet/pods/cb922f2e-feb3-4249-a4a4-052ae2f4c6a0/volumes"
Nov 03 20:52:12.123147 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.112485    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f37b9dfa-401a-476d-9387-50a7f6cd1d3f" path="/var/lib/kubelet/pods/f37b9dfa-401a-476d-9387-50a7f6cd1d3f/volumes"
Nov 03 20:52:12.191966 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.191922    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.192225 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.192203    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.192319 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.192303    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8g2m\" (UniqueName: \"kubernetes.io/projected/a22f1f1a-d59b-476d-8474-d6d611be0d64-kube-api-access-m8g2m\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.192875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.192851    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.194971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.194930    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.239873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.239805    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z" podStartSLOduration=2.2397879 podStartE2EDuration="2.2397879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:12.233294207 +0000 UTC m=+1372.237252956" watchObservedRunningTime="2026-11-03 20:52:12.2397879 +0000 UTC m=+1372.243746624"
Nov 03 20:52:12.246632 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.246593    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8g2m\" (UniqueName: \"kubernetes.io/projected/a22f1f1a-d59b-476d-8474-d6d611be0d64-kube-api-access-m8g2m\") pod \"maas-api-revoke-sub-978212fb-l5r4k\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.332378 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.332341    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:12.466085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.466044    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"]
Nov 03 20:52:12.472273 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.472242    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.522199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.522159    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"]
Nov 03 20:52:12.599032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.598928    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.599032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.598990    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqzd\" (UniqueName: \"kubernetes.io/projected/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-kube-api-access-4jqzd\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.599300 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.599175    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.700741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.700688    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.700957 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.700748    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqzd\" (UniqueName: \"kubernetes.io/projected/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-kube-api-access-4jqzd\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.700957 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.700848    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.701545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.701524    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.703590 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.703568    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.722546 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.722519    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"]
Nov 03 20:52:12.725012 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:12.724979    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22f1f1a_d59b_476d_8474_d6d611be0d64.slice/crio-77280152ca018e31b980b2567077d18ffb4e8913145a86cd29dabca59160606e WatchSource:0}: Error finding container 77280152ca018e31b980b2567077d18ffb4e8913145a86cd29dabca59160606e: Status 404 returned error can't find the container with id 77280152ca018e31b980b2567077d18ffb4e8913145a86cd29dabca59160606e
Nov 03 20:52:12.754662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.754632    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqzd\" (UniqueName: \"kubernetes.io/projected/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-kube-api-access-4jqzd\") pod \"maas-api-revoke-sub-3f9daf6c-gc74h\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.791365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.791332    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:12.995156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:12.995123    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"]
Nov 03 20:52:13.001615 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.001595    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.056582 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.056548    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"]
Nov 03 20:52:13.103820 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.103781    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k" event={"ID":"a22f1f1a-d59b-476d-8474-d6d611be0d64","Type":"ContainerStarted","Data":"77280152ca018e31b980b2567077d18ffb4e8913145a86cd29dabca59160606e"}
Nov 03 20:52:13.105514 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.105489    5196 generic.go:356] "Generic (PLEG): container finished" podID="86a76380-07a5-487e-a82e-23dd52b7cfd5" containerID="4962933513b7cd43d2a22ff8d411c0b79ad1f80d0a5285a5496a7d378010f80d" exitCode=0
Nov 03 20:52:13.105628 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.105570    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z" event={"ID":"86a76380-07a5-487e-a82e-23dd52b7cfd5","Type":"ContainerDied","Data":"4962933513b7cd43d2a22ff8d411c0b79ad1f80d0a5285a5496a7d378010f80d"}
Nov 03 20:52:13.105952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.105924    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.106180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.106167    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4brp\" (UniqueName: \"kubernetes.io/projected/7c79042c-40e8-463b-8033-73ccfb31f01a-kube-api-access-f4brp\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.106237 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.106222    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.182909 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.182772    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"]
Nov 03 20:52:13.186264 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:13.186238    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7082949e_6d4d_4b6f_8e31_eae5d192f2f4.slice/crio-9a72ee53d1e8adffe9f0a647ba0294dd21316f997ca1118851e3f89d2f6271be WatchSource:0}: Error finding container 9a72ee53d1e8adffe9f0a647ba0294dd21316f997ca1118851e3f89d2f6271be: Status 404 returned error can't find the container with id 9a72ee53d1e8adffe9f0a647ba0294dd21316f997ca1118851e3f89d2f6271be
Nov 03 20:52:13.208175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.208062    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4brp\" (UniqueName: \"kubernetes.io/projected/7c79042c-40e8-463b-8033-73ccfb31f01a-kube-api-access-f4brp\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.208175 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.208121    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.208347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.208184    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.209074 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.209037    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.210867 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.210843    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.261877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.261846    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4brp\" (UniqueName: \"kubernetes.io/projected/7c79042c-40e8-463b-8033-73ccfb31f01a-kube-api-access-f4brp\") pod \"maas-api-revoke-sub-de4902ea-gtcvv\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.316733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.316695    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:13.801524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:13.801491    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"]
Nov 03 20:52:13.802216 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:13.802186    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c79042c_40e8_463b_8033_73ccfb31f01a.slice/crio-f2bd6fb277185f30acda36a38cee086261c34f9193df313453eb78e2e15fce40 WatchSource:0}: Error finding container f2bd6fb277185f30acda36a38cee086261c34f9193df313453eb78e2e15fce40: Status 404 returned error can't find the container with id f2bd6fb277185f30acda36a38cee086261c34f9193df313453eb78e2e15fce40
Nov 03 20:52:14.046907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.046870    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"]
Nov 03 20:52:14.053726 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.053628    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d53323-vqm9h"]
Nov 03 20:52:14.110885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.110852    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f49733fd-37e2-4a37-9066-f76314873be0" path="/var/lib/kubelet/pods/f49733fd-37e2-4a37-9066-f76314873be0/volumes"
Nov 03 20:52:14.116580 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.116543    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv" event={"ID":"7c79042c-40e8-463b-8033-73ccfb31f01a","Type":"ContainerStarted","Data":"f2bd6fb277185f30acda36a38cee086261c34f9193df313453eb78e2e15fce40"}
Nov 03 20:52:14.118627 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.118597    5196 generic.go:356] "Generic (PLEG): container finished" podID="a22f1f1a-d59b-476d-8474-d6d611be0d64" containerID="8c84c47e9c162ccb5b56eca6151aba62f1b2f19f96321249556edae434d369d6" exitCode=0
Nov 03 20:52:14.118818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.118799    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k" event={"ID":"a22f1f1a-d59b-476d-8474-d6d611be0d64","Type":"ContainerDied","Data":"8c84c47e9c162ccb5b56eca6151aba62f1b2f19f96321249556edae434d369d6"}
Nov 03 20:52:14.120771 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.120745    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h" event={"ID":"7082949e-6d4d-4b6f-8e31-eae5d192f2f4","Type":"ContainerStarted","Data":"c14da6b8a40ad4790cc986e62f2671e5dcc8252ddadf41095854746f994bfc30"}
Nov 03 20:52:14.120888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.120794    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h" event={"ID":"7082949e-6d4d-4b6f-8e31-eae5d192f2f4","Type":"ContainerStarted","Data":"9a72ee53d1e8adffe9f0a647ba0294dd21316f997ca1118851e3f89d2f6271be"}
Nov 03 20:52:14.330716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.330692    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:14.339754 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.339696    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h" podStartSLOduration=2.339681166 podStartE2EDuration="2.339681166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:14.334033472 +0000 UTC m=+1374.337992195" watchObservedRunningTime="2026-11-03 20:52:14.339681166 +0000 UTC m=+1374.343639890"
Nov 03 20:52:14.435636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.435595    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-xks-ca\") pod \"86a76380-07a5-487e-a82e-23dd52b7cfd5\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") "
Nov 03 20:52:14.435828 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.435748    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a76380-07a5-487e-a82e-23dd52b7cfd5-kube-api-access-9w9wj\" (UniqueName: \"kubernetes.io/projected/86a76380-07a5-487e-a82e-23dd52b7cfd5-kube-api-access-9w9wj\") pod \"86a76380-07a5-487e-a82e-23dd52b7cfd5\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") "
Nov 03 20:52:14.436027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.436009    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-openshift-ca\") pod \"86a76380-07a5-487e-a82e-23dd52b7cfd5\" (UID: \"86a76380-07a5-487e-a82e-23dd52b7cfd5\") "
Nov 03 20:52:14.436424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.436400    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-openshift-ca" pod "86a76380-07a5-487e-a82e-23dd52b7cfd5" (UID: "86a76380-07a5-487e-a82e-23dd52b7cfd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:14.436619 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.436591    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:14.439105 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.439064    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-xks-ca" pod "86a76380-07a5-487e-a82e-23dd52b7cfd5" (UID: "86a76380-07a5-487e-a82e-23dd52b7cfd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:14.439199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.439143    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a76380-07a5-487e-a82e-23dd52b7cfd5-kube-api-access-9w9wj" pod "86a76380-07a5-487e-a82e-23dd52b7cfd5" (UID: "86a76380-07a5-487e-a82e-23dd52b7cfd5"). InnerVolumeSpecName "kube-api-access-9w9wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:14.538668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.538627    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a76380-07a5-487e-a82e-23dd52b7cfd5-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:14.538668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:14.538661    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w9wj\" (UniqueName: \"kubernetes.io/projected/86a76380-07a5-487e-a82e-23dd52b7cfd5-kube-api-access-9w9wj\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:15.134380 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.134344    5196 generic.go:356] "Generic (PLEG): container finished" podID="7c79042c-40e8-463b-8033-73ccfb31f01a" containerID="d16d7e7087361685bf019f6530ddb5eda68ad4bb4aa60be6677bf7252d15489e" exitCode=0
Nov 03 20:52:15.134954 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.134425    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv" event={"ID":"7c79042c-40e8-463b-8033-73ccfb31f01a","Type":"ContainerDied","Data":"d16d7e7087361685bf019f6530ddb5eda68ad4bb4aa60be6677bf7252d15489e"}
Nov 03 20:52:15.135978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.135956    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"
Nov 03 20:52:15.136149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.135992    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z" event={"ID":"86a76380-07a5-487e-a82e-23dd52b7cfd5","Type":"ContainerDied","Data":"c4ef055330006c838cbdbfbfb7f42d6c11817d53bf69f024c66848bdf2e28c42"}
Nov 03 20:52:15.136149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.136030    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4ef055330006c838cbdbfbfb7f42d6c11817d53bf69f024c66848bdf2e28c42"
Nov 03 20:52:15.137936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.137912    5196 generic.go:356] "Generic (PLEG): container finished" podID="7082949e-6d4d-4b6f-8e31-eae5d192f2f4" containerID="c14da6b8a40ad4790cc986e62f2671e5dcc8252ddadf41095854746f994bfc30" exitCode=0
Nov 03 20:52:15.138070 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.137999    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h" event={"ID":"7082949e-6d4d-4b6f-8e31-eae5d192f2f4","Type":"ContainerDied","Data":"c14da6b8a40ad4790cc986e62f2671e5dcc8252ddadf41095854746f994bfc30"}
Nov 03 20:52:15.332500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.332471    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:15.454035 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.454004    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22f1f1a-d59b-476d-8474-d6d611be0d64-kube-api-access-m8g2m\" (UniqueName: \"kubernetes.io/projected/a22f1f1a-d59b-476d-8474-d6d611be0d64-kube-api-access-m8g2m\") pod \"a22f1f1a-d59b-476d-8474-d6d611be0d64\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") "
Nov 03 20:52:15.454247 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.454228    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-xks-ca\") pod \"a22f1f1a-d59b-476d-8474-d6d611be0d64\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") "
Nov 03 20:52:15.454296 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.454279    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-openshift-ca\") pod \"a22f1f1a-d59b-476d-8474-d6d611be0d64\" (UID: \"a22f1f1a-d59b-476d-8474-d6d611be0d64\") "
Nov 03 20:52:15.454677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.454647    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-openshift-ca" pod "a22f1f1a-d59b-476d-8474-d6d611be0d64" (UID: "a22f1f1a-d59b-476d-8474-d6d611be0d64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:15.455020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.454997    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:15.456535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.456510    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22f1f1a-d59b-476d-8474-d6d611be0d64-kube-api-access-m8g2m" pod "a22f1f1a-d59b-476d-8474-d6d611be0d64" (UID: "a22f1f1a-d59b-476d-8474-d6d611be0d64"). InnerVolumeSpecName "kube-api-access-m8g2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:15.456535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.456524    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-xks-ca" pod "a22f1f1a-d59b-476d-8474-d6d611be0d64" (UID: "a22f1f1a-d59b-476d-8474-d6d611be0d64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:15.556358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.556259    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8g2m\" (UniqueName: \"kubernetes.io/projected/a22f1f1a-d59b-476d-8474-d6d611be0d64-kube-api-access-m8g2m\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:15.556358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.556293    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22f1f1a-d59b-476d-8474-d6d611be0d64-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:15.872968 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:15.872918    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:16.150353 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.150262    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"
Nov 03 20:52:16.150353 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.150284    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k" event={"ID":"a22f1f1a-d59b-476d-8474-d6d611be0d64","Type":"ContainerDied","Data":"77280152ca018e31b980b2567077d18ffb4e8913145a86cd29dabca59160606e"}
Nov 03 20:52:16.150353 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.150340    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77280152ca018e31b980b2567077d18ffb4e8913145a86cd29dabca59160606e"
Nov 03 20:52:16.361044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.361018    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:16.363572 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.363552    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:16.495042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.472545    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-kube-api-access-4jqzd\" (UniqueName: \"kubernetes.io/projected/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-kube-api-access-4jqzd\") pod \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") "
Nov 03 20:52:16.495042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.472672    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-openshift-ca\") pod \"7c79042c-40e8-463b-8033-73ccfb31f01a\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") "
Nov 03 20:52:16.495042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.472794    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c79042c-40e8-463b-8033-73ccfb31f01a-kube-api-access-f4brp\" (UniqueName: \"kubernetes.io/projected/7c79042c-40e8-463b-8033-73ccfb31f01a-kube-api-access-f4brp\") pod \"7c79042c-40e8-463b-8033-73ccfb31f01a\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") "
Nov 03 20:52:16.495042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.472922    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-openshift-ca\") pod \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") "
Nov 03 20:52:16.495042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.472992    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-xks-ca\") pod \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\" (UID: \"7082949e-6d4d-4b6f-8e31-eae5d192f2f4\") "
Nov 03 20:52:16.495042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.473065    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-xks-ca\") pod \"7c79042c-40e8-463b-8033-73ccfb31f01a\" (UID: \"7c79042c-40e8-463b-8033-73ccfb31f01a\") "
Nov 03 20:52:16.495042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.478423    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c79042c-40e8-463b-8033-73ccfb31f01a-kube-api-access-f4brp" pod "7c79042c-40e8-463b-8033-73ccfb31f01a" (UID: "7c79042c-40e8-463b-8033-73ccfb31f01a"). InnerVolumeSpecName "kube-api-access-f4brp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:16.495042 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.490302    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-kube-api-access-4jqzd" pod "7082949e-6d4d-4b6f-8e31-eae5d192f2f4" (UID: "7082949e-6d4d-4b6f-8e31-eae5d192f2f4"). InnerVolumeSpecName "kube-api-access-4jqzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:16.495603 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.495172    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-openshift-ca" pod "7c79042c-40e8-463b-8033-73ccfb31f01a" (UID: "7c79042c-40e8-463b-8033-73ccfb31f01a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:16.496685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.496625    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-openshift-ca" pod "7082949e-6d4d-4b6f-8e31-eae5d192f2f4" (UID: "7082949e-6d4d-4b6f-8e31-eae5d192f2f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:16.497859 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.496836    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-xks-ca" pod "7082949e-6d4d-4b6f-8e31-eae5d192f2f4" (UID: "7082949e-6d4d-4b6f-8e31-eae5d192f2f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:16.499704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.499678    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-xks-ca" pod "7c79042c-40e8-463b-8033-73ccfb31f01a" (UID: "7c79042c-40e8-463b-8033-73ccfb31f01a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:16.575222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.575182    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:16.575222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.575219    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:16.575479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.575234    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jqzd\" (UniqueName: \"kubernetes.io/projected/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-kube-api-access-4jqzd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:16.575479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.575250    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c79042c-40e8-463b-8033-73ccfb31f01a-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:16.575479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.575262    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4brp\" (UniqueName: \"kubernetes.io/projected/7c79042c-40e8-463b-8033-73ccfb31f01a-kube-api-access-f4brp\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:16.575479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:16.575273    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7082949e-6d4d-4b6f-8e31-eae5d192f2f4-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:17.160969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:17.160939    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"
Nov 03 20:52:17.160969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:17.160957    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h" event={"ID":"7082949e-6d4d-4b6f-8e31-eae5d192f2f4","Type":"ContainerDied","Data":"9a72ee53d1e8adffe9f0a647ba0294dd21316f997ca1118851e3f89d2f6271be"}
Nov 03 20:52:17.161486 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:17.161014    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a72ee53d1e8adffe9f0a647ba0294dd21316f997ca1118851e3f89d2f6271be"
Nov 03 20:52:17.162614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:17.162595    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"
Nov 03 20:52:17.162721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:17.162589    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv" event={"ID":"7c79042c-40e8-463b-8033-73ccfb31f01a","Type":"ContainerDied","Data":"f2bd6fb277185f30acda36a38cee086261c34f9193df313453eb78e2e15fce40"}
Nov 03 20:52:17.162759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:17.162723    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2bd6fb277185f30acda36a38cee086261c34f9193df313453eb78e2e15fce40"
Nov 03 20:52:18.047775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:18.047743    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"]
Nov 03 20:52:18.094236 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:18.094198    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe9b41d-lhrdp"]
Nov 03 20:52:18.113034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:18.113003    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88653563-4582-4c54-8175-e0f9d3a94bf7" path="/var/lib/kubelet/pods/88653563-4582-4c54-8175-e0f9d3a94bf7/volumes"
Nov 03 20:52:18.487787 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:18.487753    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:52:20.386733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:20.386695    5196 scope.go:122] "RemoveContainer" containerID="3cc43c4c6c9dfa18991c304ab54cd3e3549d2d130693609975425a68249d1f12"
Nov 03 20:52:20.399111 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:20.399073    5196 scope.go:122] "RemoveContainer" containerID="2a32da781c06f4ad795737054622f3c7caf2646bd7f85765d1f9d5c33f117a1c"
Nov 03 20:52:20.411487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:20.411462    5196 scope.go:122] "RemoveContainer" containerID="66ffe0f9cab6fbccda6ca0feb96c2bedadf64b64e7be591bcd8db64b550a75fc"
Nov 03 20:52:20.423684 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:20.423651    5196 scope.go:122] "RemoveContainer" containerID="16f1fd1ba6ed2e7432d561d0f6c24b1f4ec653bfdfb65f3810a27d0eb2035e02"
Nov 03 20:52:20.443337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:20.443302    5196 scope.go:122] "RemoveContainer" containerID="a3dbb0f8536b043d150b0de9b9754a9556e4d0a8f02e856b2dcc35538b590291"
Nov 03 20:52:20.462458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:20.462434    5196 scope.go:122] "RemoveContainer" containerID="eb9816f034f66da53a048fa6408a3f865ac5c890ad120e932aff377277d6a809"
Nov 03 20:52:20.658999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:20.658913    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:22.287413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.287360    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"]
Nov 03 20:52:22.288156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288138    5196 cpu_manager.go:414] "containerMap: removing container" podUID="86a76380-07a5-487e-a82e-23dd52b7cfd5" containerName="revoke-keys"
Nov 03 20:52:22.288222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288157    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a76380-07a5-487e-a82e-23dd52b7cfd5" containerName="revoke-keys"
Nov 03 20:52:22.288222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288180    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a22f1f1a-d59b-476d-8474-d6d611be0d64" containerName="revoke-keys"
Nov 03 20:52:22.288222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288185    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22f1f1a-d59b-476d-8474-d6d611be0d64" containerName="revoke-keys"
Nov 03 20:52:22.288222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288196    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7082949e-6d4d-4b6f-8e31-eae5d192f2f4" containerName="revoke-keys"
Nov 03 20:52:22.288222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288202    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7082949e-6d4d-4b6f-8e31-eae5d192f2f4" containerName="revoke-keys"
Nov 03 20:52:22.288222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288213    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7c79042c-40e8-463b-8033-73ccfb31f01a" containerName="revoke-keys"
Nov 03 20:52:22.288222 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288217    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c79042c-40e8-463b-8033-73ccfb31f01a" containerName="revoke-keys"
Nov 03 20:52:22.288445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288349    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7082949e-6d4d-4b6f-8e31-eae5d192f2f4" containerName="revoke-keys"
Nov 03 20:52:22.288445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288359    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c79042c-40e8-463b-8033-73ccfb31f01a" containerName="revoke-keys"
Nov 03 20:52:22.288445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288372    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a76380-07a5-487e-a82e-23dd52b7cfd5" containerName="revoke-keys"
Nov 03 20:52:22.288445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.288381    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22f1f1a-d59b-476d-8474-d6d611be0d64" containerName="revoke-keys"
Nov 03 20:52:22.294307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.294286    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.343548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.343512    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"]
Nov 03 20:52:22.357993 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.357948    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.358435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.358402    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.358613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.358518    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqz4\" (UniqueName: \"kubernetes.io/projected/1c210b55-ce4e-4b6f-b069-9b37c074b52f-kube-api-access-thqz4\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.384012 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.383978    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:52:22.384195 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.384045    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:52:22.459896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.459856    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.460082 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.459917    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqz4\" (UniqueName: \"kubernetes.io/projected/1c210b55-ce4e-4b6f-b069-9b37c074b52f-kube-api-access-thqz4\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.460201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.460081    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.460764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.460697    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.462674 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.462647    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.514290 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.514250    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqz4\" (UniqueName: \"kubernetes.io/projected/1c210b55-ce4e-4b6f-b069-9b37c074b52f-kube-api-access-thqz4\") pod \"maas-api-revoke-sub-3fdd0566-q5tkk\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.608894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.608866    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:22.750184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.750140    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"]
Nov 03 20:52:22.761191 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.761156    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.806371 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.806329    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"]
Nov 03 20:52:22.871733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.871650    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scxx\" (UniqueName: \"kubernetes.io/projected/9b35dcf3-37da-4854-aee6-abdb8fb71f30-kube-api-access-7scxx\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.871733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.871698    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.871944 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.871835    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.973582 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.973541    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scxx\" (UniqueName: \"kubernetes.io/projected/9b35dcf3-37da-4854-aee6-abdb8fb71f30-kube-api-access-7scxx\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.973582 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.973587    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.973845 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.973641    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.974491 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.974466    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.976260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.976235    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:22.994988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:22.994954    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"]
Nov 03 20:52:22.997810 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:22.997768    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c210b55_ce4e_4b6f_b069_9b37c074b52f.slice/crio-76404d0048d983355090c40bcb78e16e89dbc4d6f068045042a71a873e913e47 WatchSource:0}: Error finding container 76404d0048d983355090c40bcb78e16e89dbc4d6f068045042a71a873e913e47: Status 404 returned error can't find the container with id 76404d0048d983355090c40bcb78e16e89dbc4d6f068045042a71a873e913e47
Nov 03 20:52:23.027782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:23.027757    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scxx\" (UniqueName: \"kubernetes.io/projected/9b35dcf3-37da-4854-aee6-abdb8fb71f30-kube-api-access-7scxx\") pod \"maas-api-revoke-sub-e78a14bc-xw9hq\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:23.086176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:23.086146    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:23.227043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:23.227009    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk" event={"ID":"1c210b55-ce4e-4b6f-b069-9b37c074b52f","Type":"ContainerStarted","Data":"76404d0048d983355090c40bcb78e16e89dbc4d6f068045042a71a873e913e47"}
Nov 03 20:52:23.514295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:23.514267    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"]
Nov 03 20:52:23.515667 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:23.515637    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b35dcf3_37da_4854_aee6_abdb8fb71f30.slice/crio-b5de634539723408c5588f90ae2be371c896ed9e2d7c1537b8de4f464f52f3c8 WatchSource:0}: Error finding container b5de634539723408c5588f90ae2be371c896ed9e2d7c1537b8de4f464f52f3c8: Status 404 returned error can't find the container with id b5de634539723408c5588f90ae2be371c896ed9e2d7c1537b8de4f464f52f3c8
Nov 03 20:52:24.050495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.050454    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"]
Nov 03 20:52:24.057056 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.057030    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149fac51-pv6ms"]
Nov 03 20:52:24.111790 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.111682    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d06932b-717a-4def-9cfb-a4e2372cd6e6" path="/var/lib/kubelet/pods/3d06932b-717a-4def-9cfb-a4e2372cd6e6/volumes"
Nov 03 20:52:24.238325 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.238289    5196 generic.go:356] "Generic (PLEG): container finished" podID="1c210b55-ce4e-4b6f-b069-9b37c074b52f" containerID="2572add56b238b6cb18975853dbe3ab07026e2c229d3e407b738209d55aa33e3" exitCode=0
Nov 03 20:52:24.238519 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.238432    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk" event={"ID":"1c210b55-ce4e-4b6f-b069-9b37c074b52f","Type":"ContainerDied","Data":"2572add56b238b6cb18975853dbe3ab07026e2c229d3e407b738209d55aa33e3"}
Nov 03 20:52:24.240384 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.240357    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq" event={"ID":"9b35dcf3-37da-4854-aee6-abdb8fb71f30","Type":"ContainerStarted","Data":"44ec4a474ad6cd86875fddffe6efe3a8adedd9a8396015faf260713a55df6c1b"}
Nov 03 20:52:24.240498 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.240411    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq" event={"ID":"9b35dcf3-37da-4854-aee6-abdb8fb71f30","Type":"ContainerStarted","Data":"b5de634539723408c5588f90ae2be371c896ed9e2d7c1537b8de4f464f52f3c8"}
Nov 03 20:52:24.451210 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.451078    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq" podStartSLOduration=2.451063992 podStartE2EDuration="2.451063992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:24.449612092 +0000 UTC m=+1384.453570836" watchObservedRunningTime="2026-11-03 20:52:24.451063992 +0000 UTC m=+1384.455022731"
Nov 03 20:52:24.988640 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.988599    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"]
Nov 03 20:52:24.993521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:24.993498    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.100178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.100137    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"]
Nov 03 20:52:25.113704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.113664    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"]
Nov 03 20:52:25.114551 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.114526    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.118502    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.118575    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.118624    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7hn\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-kube-api-access-bb7hn\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.118706    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.118750    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.118879    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.118965    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.119016    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.120794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.119157    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.131501 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.131466    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-896fc70e-r22bp"]
Nov 03 20:52:25.153601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.153525    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"]
Nov 03 20:52:25.167884 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.167843    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"]
Nov 03 20:52:25.220621 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.220579    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.220810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.220634    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.220810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.220671    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.220810 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.220714    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.221161 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221132    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.221251 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221219    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.221297 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221263    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.221415 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221390    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.221542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221493    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj95z\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-kube-api-access-fj95z\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.221605 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221544    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.221891 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221863    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-data\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.221999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221975    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-credential-socket\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.222335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.221608    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.222439 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.222369    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.222520 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.222486    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.222622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.222572    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.222622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.222600    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.222739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.222632    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.222739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.222663    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istiod-ca-cert\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.222739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.222696    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.222955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.222929    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-certs\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.223149 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.223108    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7hn\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-kube-api-access-bb7hn\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.223217 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.223164    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.224484 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.224449    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-envoy\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.225695 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.225670    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-podinfo\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.254292 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.254211    5196 generic.go:356] "Generic (PLEG): container finished" podID="9b35dcf3-37da-4854-aee6-abdb8fb71f30" containerID="44ec4a474ad6cd86875fddffe6efe3a8adedd9a8396015faf260713a55df6c1b" exitCode=0
Nov 03 20:52:25.254456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.254301    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq" event={"ID":"9b35dcf3-37da-4854-aee6-abdb8fb71f30","Type":"ContainerDied","Data":"44ec4a474ad6cd86875fddffe6efe3a8adedd9a8396015faf260713a55df6c1b"}
Nov 03 20:52:25.276795 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.276763    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-token\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.277590 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.277569    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7hn\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-kube-api-access-bb7hn\") pod \"e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") " pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.319145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.319052    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:25.324206 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324177    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj95z\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-kube-api-access-fj95z\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324341 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324262    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324346    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324370    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324513 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324472    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324570 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324558    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324625 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324599    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324663    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324732 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324704    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324795 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324767    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-credential-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.324851 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.324809    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-certs\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.325215 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.325190    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istiod-ca-cert\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.325292 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.325211    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-data\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.325442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.325418    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-socket\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.328865 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.328836    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-podinfo\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.333299 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.332367    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-envoy\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.379885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.379421    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-token\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.379885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.379831    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj95z\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-kube-api-access-fj95z\") pod \"e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") " pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.384716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.384468    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:25.455506 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.455470    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:25.456157 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.456136    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:25.536412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.535009    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-xks-ca\") pod \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") "
Nov 03 20:52:25.536412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.535082    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c210b55-ce4e-4b6f-b069-9b37c074b52f-kube-api-access-thqz4\" (UniqueName: \"kubernetes.io/projected/1c210b55-ce4e-4b6f-b069-9b37c074b52f-kube-api-access-thqz4\") pod \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") "
Nov 03 20:52:25.536412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.535281    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-openshift-ca\") pod \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\" (UID: \"1c210b55-ce4e-4b6f-b069-9b37c074b52f\") "
Nov 03 20:52:25.536707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.536597    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-openshift-ca" pod "1c210b55-ce4e-4b6f-b069-9b37c074b52f" (UID: "1c210b55-ce4e-4b6f-b069-9b37c074b52f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:25.542646 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.542605    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-xks-ca" pod "1c210b55-ce4e-4b6f-b069-9b37c074b52f" (UID: "1c210b55-ce4e-4b6f-b069-9b37c074b52f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:25.556083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.547204    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c210b55-ce4e-4b6f-b069-9b37c074b52f-kube-api-access-thqz4" pod "1c210b55-ce4e-4b6f-b069-9b37c074b52f" (UID: "1c210b55-ce4e-4b6f-b069-9b37c074b52f"). InnerVolumeSpecName "kube-api-access-thqz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:25.637720 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.637679    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:25.637720 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.637719    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c210b55-ce4e-4b6f-b069-9b37c074b52f-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:25.637926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.637733    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thqz4\" (UniqueName: \"kubernetes.io/projected/1c210b55-ce4e-4b6f-b069-9b37c074b52f-kube-api-access-thqz4\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:25.783743 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.783718    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"]
Nov 03 20:52:25.786963 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:25.786895    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb74f130_9b42_4f99_b7ee_59a64b93d4da.slice/crio-90476724bcab730946f0ed72f892c1770cc31f9a55313649f9026ab303f185b5 WatchSource:0}: Error finding container 90476724bcab730946f0ed72f892c1770cc31f9a55313649f9026ab303f185b5: Status 404 returned error can't find the container with id 90476724bcab730946f0ed72f892c1770cc31f9a55313649f9026ab303f185b5
Nov 03 20:52:25.790346 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.790302    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:25.790452 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.790382    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:25.790452 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.790412    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:25.873728 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.873686    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"]
Nov 03 20:52:25.876951 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:25.876909    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6f4c3f_8932_4cdf_a1b5_fd829035d788.slice/crio-a77dbca93d14eae8d91d0bedfc6207da6a3542b52c2c61ee67720c72b836809c WatchSource:0}: Error finding container a77dbca93d14eae8d91d0bedfc6207da6a3542b52c2c61ee67720c72b836809c: Status 404 returned error can't find the container with id a77dbca93d14eae8d91d0bedfc6207da6a3542b52c2c61ee67720c72b836809c
Nov 03 20:52:25.879683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.879645    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:25.879764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.879723    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:25.879764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:25.879758    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:26.115333 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.115255    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376dc5c5-3f5c-43f9-826c-aeb3298fd47a" path="/var/lib/kubelet/pods/376dc5c5-3f5c-43f9-826c-aeb3298fd47a/volumes"
Nov 03 20:52:26.269861 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.269825    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" event={"ID":"5c6f4c3f-8932-4cdf-a1b5-fd829035d788","Type":"ContainerStarted","Data":"1af1a0f3ada2b5eafdc3b5f2b67178f3634ef4999b64d8fc1d7c6cab583aec9e"}
Nov 03 20:52:26.270047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.269889    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" event={"ID":"5c6f4c3f-8932-4cdf-a1b5-fd829035d788","Type":"ContainerStarted","Data":"a77dbca93d14eae8d91d0bedfc6207da6a3542b52c2c61ee67720c72b836809c"}
Nov 03 20:52:26.272661 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.272625    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" event={"ID":"cb74f130-9b42-4f99-b7ee-59a64b93d4da","Type":"ContainerStarted","Data":"b613846a3458598d2d3d5d78b863e326656319646c823619d888fc34456ed227"}
Nov 03 20:52:26.272799 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.272679    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" event={"ID":"cb74f130-9b42-4f99-b7ee-59a64b93d4da","Type":"ContainerStarted","Data":"90476724bcab730946f0ed72f892c1770cc31f9a55313649f9026ab303f185b5"}
Nov 03 20:52:26.274715 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.274684    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"
Nov 03 20:52:26.274868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.274732    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk" event={"ID":"1c210b55-ce4e-4b6f-b069-9b37c074b52f","Type":"ContainerDied","Data":"76404d0048d983355090c40bcb78e16e89dbc4d6f068045042a71a873e913e47"}
Nov 03 20:52:26.274868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.274769    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76404d0048d983355090c40bcb78e16e89dbc4d6f068045042a71a873e913e47"
Nov 03 20:52:26.321998 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.320272    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:26.322715 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.322679    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:52:26.322875 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.322740    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 20:52:26.416164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.413041    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podStartSLOduration=1.41302546 podStartE2EDuration="1.41302546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:26.407169172 +0000 UTC m=+1386.411127925" watchObservedRunningTime="2026-11-03 20:52:26.41302546 +0000 UTC m=+1386.416984183"
Nov 03 20:52:26.460011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.459918    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:26.462583 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.462264    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:52:26.462583 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.462323    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 20:52:26.491885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.491844    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7295-ea7c3e"]
Nov 03 20:52:26.492823 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.492788    5196 cpu_manager.go:414] "containerMap: removing container" podUID="1c210b55-ce4e-4b6f-b069-9b37c074b52f" containerName="revoke-keys"
Nov 03 20:52:26.492988 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.492975    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c210b55-ce4e-4b6f-b069-9b37c074b52f" containerName="revoke-keys"
Nov 03 20:52:26.493289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.493277    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c210b55-ce4e-4b6f-b069-9b37c074b52f" containerName="revoke-keys"
Nov 03 20:52:26.493428 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.493407    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:26.497229 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.497208    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:26.509556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.509528    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-ea7c3e"]
Nov 03 20:52:26.569980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.559492    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-24vlg\""
Nov 03 20:52:26.569980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.568684    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b35dcf3-37da-4854-aee6-abdb8fb71f30-kube-api-access-7scxx\" (UniqueName: \"kubernetes.io/projected/9b35dcf3-37da-4854-aee6-abdb8fb71f30-kube-api-access-7scxx\") pod \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") "
Nov 03 20:52:26.569980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.568857    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-openshift-ca\") pod \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") "
Nov 03 20:52:26.569980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.568936    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-xks-ca\") pod \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\" (UID: \"9b35dcf3-37da-4854-aee6-abdb8fb71f30\") "
Nov 03 20:52:26.569980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.569608    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68875\" (UniqueName: \"kubernetes.io/projected/9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9-kube-api-access-68875\") pod \"test-curl-7295-ea7c3e\" (UID: \"9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9\") " pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:26.570670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.570637    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-openshift-ca" pod "9b35dcf3-37da-4854-aee6-abdb8fb71f30" (UID: "9b35dcf3-37da-4854-aee6-abdb8fb71f30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:26.572489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.572459    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b35dcf3-37da-4854-aee6-abdb8fb71f30-kube-api-access-7scxx" pod "9b35dcf3-37da-4854-aee6-abdb8fb71f30" (UID: "9b35dcf3-37da-4854-aee6-abdb8fb71f30"). InnerVolumeSpecName "kube-api-access-7scxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:26.573343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.573308    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-xks-ca" pod "9b35dcf3-37da-4854-aee6-abdb8fb71f30" (UID: "9b35dcf3-37da-4854-aee6-abdb8fb71f30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:26.575900 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.575846    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podStartSLOduration=2.575826646 podStartE2EDuration="2.575826646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:26.573826382 +0000 UTC m=+1386.577785118" watchObservedRunningTime="2026-11-03 20:52:26.575826646 +0000 UTC m=+1386.579785370"
Nov 03 20:52:26.696965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.696138    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68875\" (UniqueName: \"kubernetes.io/projected/9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9-kube-api-access-68875\") pod \"test-curl-7295-ea7c3e\" (UID: \"9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9\") " pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:26.696965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.696600    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7scxx\" (UniqueName: \"kubernetes.io/projected/9b35dcf3-37da-4854-aee6-abdb8fb71f30-kube-api-access-7scxx\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:26.696965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.696627    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:26.696965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.696642    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b35dcf3-37da-4854-aee6-abdb8fb71f30-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:26.795842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.795788    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68875\" (UniqueName: \"kubernetes.io/projected/9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9-kube-api-access-68875\") pod \"test-curl-7295-ea7c3e\" (UID: \"9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9\") " pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:26.814848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.814813    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:26.821178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.821138    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7292-2c55a4"]
Nov 03 20:52:26.821916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.821887    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9b35dcf3-37da-4854-aee6-abdb8fb71f30" containerName="revoke-keys"
Nov 03 20:52:26.821916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.821904    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b35dcf3-37da-4854-aee6-abdb8fb71f30" containerName="revoke-keys"
Nov 03 20:52:26.822084 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.822063    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b35dcf3-37da-4854-aee6-abdb8fb71f30" containerName="revoke-keys"
Nov 03 20:52:26.826221 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.826194    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:26.971397 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:26.971286    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-2c55a4"]
Nov 03 20:52:27.016085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.016047    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2672\" (UniqueName: \"kubernetes.io/projected/4cbc5b46-2769-4f39-8ef8-b370eef7dfd4-kube-api-access-f2672\") pod \"test-curl-7292-2c55a4\" (UID: \"4cbc5b46-2769-4f39-8ef8-b370eef7dfd4\") " pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:27.130208 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.123977    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2672\" (UniqueName: \"kubernetes.io/projected/4cbc5b46-2769-4f39-8ef8-b370eef7dfd4-kube-api-access-f2672\") pod \"test-curl-7292-2c55a4\" (UID: \"4cbc5b46-2769-4f39-8ef8-b370eef7dfd4\") " pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:27.194545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.194502    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2672\" (UniqueName: \"kubernetes.io/projected/4cbc5b46-2769-4f39-8ef8-b370eef7dfd4-kube-api-access-f2672\") pod \"test-curl-7292-2c55a4\" (UID: \"4cbc5b46-2769-4f39-8ef8-b370eef7dfd4\") " pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:27.289863 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.289774    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"]
Nov 03 20:52:27.294906 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.294865    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:27.296548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.296485    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.303434 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.303399    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"
Nov 03 20:52:27.303973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.303943    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq" event={"ID":"9b35dcf3-37da-4854-aee6-abdb8fb71f30","Type":"ContainerDied","Data":"b5de634539723408c5588f90ae2be371c896ed9e2d7c1537b8de4f464f52f3c8"}
Nov 03 20:52:27.304135 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.303995    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5de634539723408c5588f90ae2be371c896ed9e2d7c1537b8de4f464f52f3c8"
Nov 03 20:52:27.309972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.309946    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"]
Nov 03 20:52:27.316661 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.316636    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.320417 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.320389    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:52:27.320534 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.320433    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 20:52:27.328465 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.328435    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7295-ea7c3e"]
Nov 03 20:52:27.330855 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:27.330827    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a67d6a6_f4a3_4cd6_9c30_6d21f62e44b9.slice/crio-64e250208634331867716248ce096383484bc7a40581904c6c59e01e7b46c88c WatchSource:0}: Error finding container 64e250208634331867716248ce096383484bc7a40581904c6c59e01e7b46c88c: Status 404 returned error can't find the container with id 64e250208634331867716248ce096383484bc7a40581904c6c59e01e7b46c88c
Nov 03 20:52:27.331344 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331319    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.331401 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331386    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.331460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331443    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6fwt\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-kube-api-access-n6fwt\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.331568 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331553    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.331614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331600    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.331667 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331654    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.331721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331704    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.331803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331757    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.331842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331814    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.331981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.331964    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.332035 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.332020    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5671dd3-4835-4f27-93db-f34fcca38ca4-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.332135 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.332118    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.332316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.332156    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.332316 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.332300    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.332426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.332356    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.332426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.332387    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k5m4\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-kube-api-access-7k5m4\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.332492 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.332457    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.332527 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.332504    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.385123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.368973    5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:52:27.420135 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.420047    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"]
Nov 03 20:52:27.431683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.431648    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"]
Nov 03 20:52:27.433894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.433867    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.433973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.433921    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5671dd3-4835-4f27-93db-f34fcca38ca4-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.434018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.433970    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.434018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.433997    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.434083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434074    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.434182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434160    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.434241 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434199    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k5m4\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-kube-api-access-7k5m4\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434269    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434331    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434388    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434427    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434464    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6fwt\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-kube-api-access-n6fwt\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434517    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434545    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434594    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434629    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434659    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.434700    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.438909    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5671dd3-4835-4f27-93db-f34fcca38ca4-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.448164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.442424    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.486463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.454538    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.486463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.454811    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.486463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.459454    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.486463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.484592    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-certs\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.486463 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.485518    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.492435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.490452    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-data\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.492435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.492174    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:52:27.492435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.492256    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 20:52:27.492977 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.492914    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istiod-ca-cert\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.494473 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.494405    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-credential-socket\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.500911 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.500856    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.501399 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.501373    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-podinfo\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.501940 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.501912    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.503227 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.503201    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-envoy\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.509552 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.509495    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6fwt\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-kube-api-access-n6fwt\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.592021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.591457    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k5m4\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-kube-api-access-7k5m4\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.595893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.595854    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.596252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.596199    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-token\") pod \"e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") " pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.622610 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.622337    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:27.694276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.694041    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:27.799482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:27.799057    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-2c55a4"]
Nov 03 20:52:27.817449 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:27.817394    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cbc5b46_2769_4f39_8ef8_b370eef7dfd4.slice/crio-8c3abcb64e71130b0516dcf6cf1d7e323b0c51d5f5e211bdb1b661dac8c5e06e WatchSource:0}: Error finding container 8c3abcb64e71130b0516dcf6cf1d7e323b0c51d5f5e211bdb1b661dac8c5e06e: Status 404 returned error can't find the container with id 8c3abcb64e71130b0516dcf6cf1d7e323b0c51d5f5e211bdb1b661dac8c5e06e
Nov 03 20:52:28.199435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.199407    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"]
Nov 03 20:52:28.208282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.208253    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"]
Nov 03 20:52:28.208942 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.208720    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:28.208942 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.208799    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:28.208942 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.208837    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:28.209439 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:28.209408    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924eacd5_f3cd_47b4_8607_0f5dc26c4747.slice/crio-13e8a7e9a8a3a95ed4efeed1d0197f3cf28de6123e45b040029a11d7a864403b WatchSource:0}: Error finding container 13e8a7e9a8a3a95ed4efeed1d0197f3cf28de6123e45b040029a11d7a864403b: Status 404 returned error can't find the container with id 13e8a7e9a8a3a95ed4efeed1d0197f3cf28de6123e45b040029a11d7a864403b
Nov 03 20:52:28.212605 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.212572    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:28.212707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.212651    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:28.212707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.212698    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:52:28.315252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.315214    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-ea7c3e" event={"ID":"9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9","Type":"ContainerStarted","Data":"64e250208634331867716248ce096383484bc7a40581904c6c59e01e7b46c88c"}
Nov 03 20:52:28.317728 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.317691    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" event={"ID":"924eacd5-f3cd-47b4-8607-0f5dc26c4747","Type":"ContainerStarted","Data":"13e8a7e9a8a3a95ed4efeed1d0197f3cf28de6123e45b040029a11d7a864403b"}
Nov 03 20:52:28.319339 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.319311    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-2c55a4" event={"ID":"4cbc5b46-2769-4f39-8ef8-b370eef7dfd4","Type":"ContainerStarted","Data":"8c3abcb64e71130b0516dcf6cf1d7e323b0c51d5f5e211bdb1b661dac8c5e06e"}
Nov 03 20:52:28.319613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.319591    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:52:28.319701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.319639    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 20:52:28.321616 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.321589    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" event={"ID":"f5671dd3-4835-4f27-93db-f34fcca38ca4","Type":"ContainerStarted","Data":"e3a7dd2b5fb58153acdd47d5e6d437fbeb0df3bfe66c12e6c45af53e15709e4e"}
Nov 03 20:52:28.458695 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.458496    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:52:28.458695 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:28.458568    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 20:52:29.321001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.320531    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:52:29.321001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.320598    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 20:52:29.440160 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.435536    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" event={"ID":"f5671dd3-4835-4f27-93db-f34fcca38ca4","Type":"ContainerStarted","Data":"0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a"}
Nov 03 20:52:29.465120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.446486    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" event={"ID":"924eacd5-f3cd-47b4-8607-0f5dc26c4747","Type":"ContainerStarted","Data":"f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8"}
Nov 03 20:52:29.491122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.489241    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:52:29.491122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.489312    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 20:52:29.637260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.637168    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:29.652117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.637707    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:52:29.652117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.637764    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 20:52:29.684119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.683655    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podStartSLOduration=2.683630182 podStartE2EDuration="2.683630182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:29.613639962 +0000 UTC m=+1389.617598714" watchObservedRunningTime="2026-11-03 20:52:29.683630182 +0000 UTC m=+1389.687588906"
Nov 03 20:52:29.768084 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.722206    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:29.768084 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.728258    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:52:29.768084 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.728321    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 20:52:29.836235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:29.836153    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podStartSLOduration=2.836127113 podStartE2EDuration="2.836127113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:29.745468213 +0000 UTC m=+1389.749426958" watchObservedRunningTime="2026-11-03 20:52:29.836127113 +0000 UTC m=+1389.840085837"
Nov 03 20:52:30.298594 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.265808    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"]
Nov 03 20:52:30.357293 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.357257    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:52:30.389376 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.389317    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 20:52:30.390045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.358934    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3607f42c-gntbz"]
Nov 03 20:52:30.456374 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.455843    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:52:30.456374 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.455900    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 20:52:30.570606 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.566954    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:30.626704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.626660    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:52:30.626957 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.626928    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 20:52:30.722506 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.722469    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:30.758540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.733063    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:30.776296 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.776255    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:52:30.776592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.776562    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 20:52:30.800843 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.800800    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:30.923281 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:30.920561    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:31.057121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.056257    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"]
Nov 03 20:52:31.108153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.095201    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:31.108153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.095647    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 20:52:31.175180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.174246    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.180477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.178031    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:31.210157 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.205904    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded"
Nov 03 20:52:31.290134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.289265    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:31.290134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.290044    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": context deadline exceeded"
Nov 03 20:52:31.445953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.317809    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:31.445953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.334035    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:52:31.445953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.334294    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:52:31.445953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.390442    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:52:31.445953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.390502    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 20:52:31.445953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.417736    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jn7\" (UniqueName: \"kubernetes.io/projected/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-kube-api-access-s4jn7\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.445953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.417858    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.445953 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.417921    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.486643 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.459023    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:52:31.486643 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.459086    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 20:52:31.544122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.540197    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jn7\" (UniqueName: \"kubernetes.io/projected/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-kube-api-access-s4jn7\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.583130 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.569671    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.600116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.583407    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.601139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.589022    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.661362 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.624077    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:52:31.661362 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.624154    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 20:52:31.757176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.688262    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"]
Nov 03 20:52:31.757176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.715684    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:52:31.757176 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.715738    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 20:52:31.771296 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.759389    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:31.771296 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.765655    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:52:31.780585 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.780514    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.825131 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.802305    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:31.825131 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.802779    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:52:31.863116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.851336    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jn7\" (UniqueName: \"kubernetes.io/projected/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-kube-api-access-s4jn7\") pod \"maas-api-revoke-sub-96692c87-f6bt6\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:31.892121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:31.878401    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:32.196122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.190921    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 03 20:52:32.196122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.191362    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/health\": context deadline exceeded"
Nov 03 20:52:32.272121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.270572    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 20:52:32.272121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.270926    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": context deadline exceeded"
Nov 03 20:52:32.297122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.285235    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:32.310122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.307204    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:32.310122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.307627    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:32.596672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.595989    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:32.597021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.596895    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:32.659759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.597176    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:52:32.659759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.597214    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 20:52:32.659759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.597442    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:52:32.659759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.597472    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 20:52:32.704405 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.704365    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:52:32.704668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.704640    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 20:52:32.737193 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.726359    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:52:32.737521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.737483    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 20:52:32.816555 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.816505    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:32.849217 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:32.849077    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:33.112189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.071290    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49799c6c-7fc4-4fd1-8f5d-861a0759d6b1" path="/var/lib/kubelet/pods/49799c6c-7fc4-4fd1-8f5d-861a0759d6b1/volumes"
Nov 03 20:52:33.112189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.072025    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:52:33.249473 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.249426    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:52:33.281321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.281281    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:33.281888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.249919    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 03 20:52:33.392302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.328421    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": context deadline exceeded"
Nov 03 20:52:33.406630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.328457    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 03 20:52:33.423228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.329675    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded"
Nov 03 20:52:33.424856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.330209    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:52:33.452300 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.330267    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:52:33.452781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.452748    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:52:33.453005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.452993    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:52:33.453282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.453266    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:52:33.453506 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.453493    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:52:33.453649 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.453637    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:52:33.453855 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.453844    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:52:33.454044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.454033    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:52:33.485077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.432217    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 20:52:33.555396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.519149    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:52:33.555396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.525914    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:52:33.555396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.525981    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 03 20:52:33.700171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.683292    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": context deadline exceeded"
Nov 03 20:52:33.700171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.683649    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded"
Nov 03 20:52:33.700171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.687220    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:52:33.700171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.687269    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 20:52:33.716540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.716491    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:52:33.716823 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.716786    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 20:52:33.852117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.834177    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:33.932153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:33.932087    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:52:34.020655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.002388    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:34.032613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.022311    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:52:34.389343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.386556    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 20:52:34.400201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.400155    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:52:34.653172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.652295    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:52:34.653172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.652352    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 20:52:34.739164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.738131    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:52:34.739164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.738181    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 20:52:34.753708 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.753661    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:34.848783 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.848744    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:34.942613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.867356    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:34.943123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:34.942180    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:35.101123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.085740    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:35.101746 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.101695    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": context deadline exceeded"
Nov 03 20:52:35.122203 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.122166    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:35.123746 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.102764    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:35.123746 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.123023    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-2c55a4" event={"ID":"4cbc5b46-2769-4f39-8ef8-b370eef7dfd4","Type":"ContainerStarted","Data":"720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d"}
Nov 03 20:52:35.124188 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.124169    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 20:52:35.173120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.161415    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7295-ea7c3e" event={"ID":"9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9","Type":"ContainerStarted","Data":"1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37"}
Nov 03 20:52:35.202183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.174339    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"]
Nov 03 20:52:35.314594 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.305079    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:52:35.329359 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.320394    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:52:35.397130 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.395116    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:35.398265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.398240    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:52:35.398383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.398319    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:35.398441 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.398382    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6" event={"ID":"4f48dad2-b50a-4b23-be53-6c2e082cc2fb","Type":"ContainerStarted","Data":"d27a4c483d6f58a913f91cb6cd53ce5c46e703ea3be83eb47a4c480fafadd60c"}
Nov 03 20:52:35.399721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.399698    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:35.402039 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.402019    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 20:52:35.423993 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.421829    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"]
Nov 03 20:52:35.493218 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.491511    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c08165-k77w2"]
Nov 03 20:52:35.502636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.502604    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:52:35.627569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.627532    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:52:35.627860 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.627820    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 20:52:35.704156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.699423    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:52:35.704156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.699482    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 20:52:35.813850 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.813728    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7295-ea7c3e" podStartSLOduration=6.734507221 podStartE2EDuration="9.813709328s" totalImagesPullingTime="3.079202107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:35.761668585 +0000 UTC m=+1395.765627308" watchObservedRunningTime="2026-11-03 20:52:35.813709328 +0000 UTC m=+1395.817668046"
Nov 03 20:52:35.923143 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.921247    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"]
Nov 03 20:52:35.932544 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.932513    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:35.972078 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.972048    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"]
Nov 03 20:52:35.991128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:35.991016    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7292-2c55a4" podStartSLOduration=7.362182095 podStartE2EDuration="9.990997174s" totalImagesPullingTime="2.628815079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:35.989984999 +0000 UTC m=+1395.993943720" watchObservedRunningTime="2026-11-03 20:52:35.990997174 +0000 UTC m=+1395.994955899"
Nov 03 20:52:36.032559 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.032495    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htscm\" (UniqueName: \"kubernetes.io/projected/2fe9721b-7852-454a-869a-94aabb82a390-kube-api-access-htscm\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.032802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.032678    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.032866 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.032848    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.052971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.052939    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"]
Nov 03 20:52:36.075238 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.075195    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b567725d-zx5dc"]
Nov 03 20:52:36.119018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.118977    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176971a3-734f-4e74-8cb2-f37fdf336d43" path="/var/lib/kubelet/pods/176971a3-734f-4e74-8cb2-f37fdf336d43/volumes"
Nov 03 20:52:36.119778 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.119760    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeae2e99-a84a-4046-a90c-8a576c38dc96" path="/var/lib/kubelet/pods/aeae2e99-a84a-4046-a90c-8a576c38dc96/volumes"
Nov 03 20:52:36.134673 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.134633    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htscm\" (UniqueName: \"kubernetes.io/projected/2fe9721b-7852-454a-869a-94aabb82a390-kube-api-access-htscm\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.134886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.134771    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.134965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.134945    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.135635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.135604    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.138366 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.138336    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.188834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.188792    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htscm\" (UniqueName: \"kubernetes.io/projected/2fe9721b-7852-454a-869a-94aabb82a390-kube-api-access-htscm\") pod \"maas-api-revoke-sub-1658d065-tsr88\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.289168 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.289086    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:36.368312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.368217    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6" event={"ID":"4f48dad2-b50a-4b23-be53-6c2e082cc2fb","Type":"ContainerStarted","Data":"d224f314767f6066ee21b5750a6420062382e488ba4aac11a796e27dce3f4ec1"}
Nov 03 20:52:36.537688 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.537632    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:36.583478 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.583394    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6" podStartSLOduration=6.583373068 podStartE2EDuration="6.583373068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:36.574490922 +0000 UTC m=+1396.578449684" watchObservedRunningTime="2026-11-03 20:52:36.583373068 +0000 UTC m=+1396.587331795"
Nov 03 20:52:36.630173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.630051    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:52:36.630173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.630133    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 20:52:36.695194 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.695161    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:52:36.695405 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.695234    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 20:52:36.752260 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:36.752231    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"]
Nov 03 20:52:36.754594 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:36.754565    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe9721b_7852_454a_869a_94aabb82a390.slice/crio-d6cb72d75e8e29ce277ec750c71bf756b010a5a35de89d9941285b3de5edb2de WatchSource:0}: Error finding container d6cb72d75e8e29ce277ec750c71bf756b010a5a35de89d9941285b3de5edb2de: Status 404 returned error can't find the container with id d6cb72d75e8e29ce277ec750c71bf756b010a5a35de89d9941285b3de5edb2de
Nov 03 20:52:37.168682 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.168633    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7295-ea7c3e"]
Nov 03 20:52:37.177056 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.177021    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7295-ea7c3e"]
Nov 03 20:52:37.385681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.385632    5196 generic.go:356] "Generic (PLEG): container finished" podID="4f48dad2-b50a-4b23-be53-6c2e082cc2fb" containerID="d224f314767f6066ee21b5750a6420062382e488ba4aac11a796e27dce3f4ec1" exitCode=0
Nov 03 20:52:37.385871 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.385850    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6" event={"ID":"4f48dad2-b50a-4b23-be53-6c2e082cc2fb","Type":"ContainerDied","Data":"d224f314767f6066ee21b5750a6420062382e488ba4aac11a796e27dce3f4ec1"}
Nov 03 20:52:37.391506 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.391473    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88" event={"ID":"2fe9721b-7852-454a-869a-94aabb82a390","Type":"ContainerStarted","Data":"1d47a9b131b55ff956bd66f7486db403f82ec57a6761d4eeb46cd9a6c8c381cb"}
Nov 03 20:52:37.391776 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.391751    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88" event={"ID":"2fe9721b-7852-454a-869a-94aabb82a390","Type":"ContainerStarted","Data":"d6cb72d75e8e29ce277ec750c71bf756b010a5a35de89d9941285b3de5edb2de"}
Nov 03 20:52:37.391900 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.391520    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7295-ea7c3e" podUID="9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" containerName="test-curl-7295-ea7c3e" containerID="cri-o://1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37" gracePeriod=2
Nov 03 20:52:37.440931 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.440889    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7292-2c55a4"]
Nov 03 20:52:37.441379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.441345    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7292-2c55a4" podUID="4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" containerName="test-curl-7292-2c55a4" containerID="cri-o://720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d" gracePeriod=2
Nov 03 20:52:37.460428 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.460388    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7292-2c55a4"]
Nov 03 20:52:37.552181 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:37.551983    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7292-2c55a4\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:37.661020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.660166    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88" podStartSLOduration=2.660149393 podStartE2EDuration="2.660149393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:37.658762369 +0000 UTC m=+1397.662721095" watchObservedRunningTime="2026-11-03 20:52:37.660149393 +0000 UTC m=+1397.664108117"
Nov 03 20:52:37.706724 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:37.706673    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-ea7c3e\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:37.919187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.919153    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:37.919873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.919843    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:37.921192 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.921163    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 20:52:37.941123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.941073    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:37.941797 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.941772    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:37.943512 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:37.943474    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 20:52:38.077532 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:38.077488    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7295-ea7c3e\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:38.125583 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:38.125540    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7292-2c55a4\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:38.203889 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.203800    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"]
Nov 03 20:52:38.204337 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.204037    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" containerID="cri-o://6742fc52080a7f35a474eaea9e7e4fceb139c500feba733865fd1516cdbef08f" gracePeriod=30
Nov 03 20:52:38.306505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.306468    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:38.410534 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.410502    5196 generic.go:356] "Generic (PLEG): container finished" podID="2fe9721b-7852-454a-869a-94aabb82a390" containerID="1d47a9b131b55ff956bd66f7486db403f82ec57a6761d4eeb46cd9a6c8c381cb" exitCode=0
Nov 03 20:52:38.410707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.410622    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88" event={"ID":"2fe9721b-7852-454a-869a-94aabb82a390","Type":"ContainerDied","Data":"1d47a9b131b55ff956bd66f7486db403f82ec57a6761d4eeb46cd9a6c8c381cb"}
Nov 03 20:52:38.460796 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.460713    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:38.614007 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.613918    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:38.618967 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.618939    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:38.699439 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.699404    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-xks-ca\") pod \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") "
Nov 03 20:52:38.699640 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.699605    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-openshift-ca\") pod \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") "
Nov 03 20:52:38.699713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.699670    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-kube-api-access-s4jn7\" (UniqueName: \"kubernetes.io/projected/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-kube-api-access-s4jn7\") pod \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\" (UID: \"4f48dad2-b50a-4b23-be53-6c2e082cc2fb\") "
Nov 03 20:52:38.700029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.699997    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-openshift-ca" pod "4f48dad2-b50a-4b23-be53-6c2e082cc2fb" (UID: "4f48dad2-b50a-4b23-be53-6c2e082cc2fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:38.700232 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.700202    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:38.702382 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.702348    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-xks-ca" pod "4f48dad2-b50a-4b23-be53-6c2e082cc2fb" (UID: "4f48dad2-b50a-4b23-be53-6c2e082cc2fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:38.702382 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.702376    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-kube-api-access-s4jn7" pod "4f48dad2-b50a-4b23-be53-6c2e082cc2fb" (UID: "4f48dad2-b50a-4b23-be53-6c2e082cc2fb"). InnerVolumeSpecName "kube-api-access-s4jn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:38.796256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.796216    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"]
Nov 03 20:52:38.797436 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797413    5196 cpu_manager.go:414] "containerMap: removing container" podUID="4f48dad2-b50a-4b23-be53-6c2e082cc2fb" containerName="revoke-keys"
Nov 03 20:52:38.797436 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797437    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f48dad2-b50a-4b23-be53-6c2e082cc2fb" containerName="revoke-keys"
Nov 03 20:52:38.797638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797475    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" containerName="test-curl-7295-ea7c3e"
Nov 03 20:52:38.797638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797483    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" containerName="test-curl-7295-ea7c3e"
Nov 03 20:52:38.797638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797553    5196 cpu_manager.go:414] "containerMap: removing container" podUID="4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" containerName="test-curl-7292-2c55a4"
Nov 03 20:52:38.797638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797560    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" containerName="test-curl-7292-2c55a4"
Nov 03 20:52:38.797842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797724    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" containerName="test-curl-7292-2c55a4"
Nov 03 20:52:38.797842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797738    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f48dad2-b50a-4b23-be53-6c2e082cc2fb" containerName="revoke-keys"
Nov 03 20:52:38.797842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.797751    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" containerName="test-curl-7295-ea7c3e"
Nov 03 20:52:38.802240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.802198    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:38.802240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.802228    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4jn7\" (UniqueName: \"kubernetes.io/projected/4f48dad2-b50a-4b23-be53-6c2e082cc2fb-kube-api-access-s4jn7\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:38.802493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.802466    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:38.854028 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.853962    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"]
Nov 03 20:52:38.903509 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.903473    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:38.903711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.903638    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxrn\" (UniqueName: \"kubernetes.io/projected/e198cf11-c062-47a5-a83f-b109821d5b58-kube-api-access-nxxrn\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:38.903884 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:38.903867    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:39.005286 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.005191    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:39.005474 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.005288    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:39.005474 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.005384    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxrn\" (UniqueName: \"kubernetes.io/projected/e198cf11-c062-47a5-a83f-b109821d5b58-kube-api-access-nxxrn\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:39.006032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.006002    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:39.008166 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.008145    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:39.059762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.059733    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxrn\" (UniqueName: \"kubernetes.io/projected/e198cf11-c062-47a5-a83f-b109821d5b58-kube-api-access-nxxrn\") pod \"maas-api-revoke-sub-36d42fd3-9r74q\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:39.137129 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.132184    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:39.427622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.427592    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"
Nov 03 20:52:39.428146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.427695    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6" event={"ID":"4f48dad2-b50a-4b23-be53-6c2e082cc2fb","Type":"ContainerDied","Data":"d27a4c483d6f58a913f91cb6cd53ce5c46e703ea3be83eb47a4c480fafadd60c"}
Nov 03 20:52:39.428146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.427825    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d27a4c483d6f58a913f91cb6cd53ce5c46e703ea3be83eb47a4c480fafadd60c"
Nov 03 20:52:39.636997 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.636958    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"]
Nov 03 20:52:39.711628 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.711518    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:39.786701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.786300    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:39.825128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.813376    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:39.871980 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.871956    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:39.872840 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.872819    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-openshift-ca\") pod \"2fe9721b-7852-454a-869a-94aabb82a390\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") "
Nov 03 20:52:39.873076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.873065    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe9721b-7852-454a-869a-94aabb82a390-kube-api-access-htscm\" (UniqueName: \"kubernetes.io/projected/2fe9721b-7852-454a-869a-94aabb82a390-kube-api-access-htscm\") pod \"2fe9721b-7852-454a-869a-94aabb82a390\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") "
Nov 03 20:52:39.873620 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.873229    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-openshift-ca" pod "2fe9721b-7852-454a-869a-94aabb82a390" (UID: "2fe9721b-7852-454a-869a-94aabb82a390"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:39.873779 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.873764    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-xks-ca\") pod \"2fe9721b-7852-454a-869a-94aabb82a390\" (UID: \"2fe9721b-7852-454a-869a-94aabb82a390\") "
Nov 03 20:52:39.874888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.874870    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:39.880692 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.880662    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-xks-ca" pod "2fe9721b-7852-454a-869a-94aabb82a390" (UID: "2fe9721b-7852-454a-869a-94aabb82a390"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:39.881002 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.880973    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe9721b-7852-454a-869a-94aabb82a390-kube-api-access-htscm" pod "2fe9721b-7852-454a-869a-94aabb82a390" (UID: "2fe9721b-7852-454a-869a-94aabb82a390"). InnerVolumeSpecName "kube-api-access-htscm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:39.976153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.976078    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9-kube-api-access-68875\" (UniqueName: \"kubernetes.io/projected/9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9-kube-api-access-68875\") pod \"9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9\" (UID: \"9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9\") "
Nov 03 20:52:39.976429 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.976407    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbc5b46-2769-4f39-8ef8-b370eef7dfd4-kube-api-access-f2672\" (UniqueName: \"kubernetes.io/projected/4cbc5b46-2769-4f39-8ef8-b370eef7dfd4-kube-api-access-f2672\") pod \"4cbc5b46-2769-4f39-8ef8-b370eef7dfd4\" (UID: \"4cbc5b46-2769-4f39-8ef8-b370eef7dfd4\") "
Nov 03 20:52:39.977032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.977010    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htscm\" (UniqueName: \"kubernetes.io/projected/2fe9721b-7852-454a-869a-94aabb82a390-kube-api-access-htscm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:39.977158 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.977036    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe9721b-7852-454a-869a-94aabb82a390-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:39.978619 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.978590    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9-kube-api-access-68875" pod "9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" (UID: "9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9"). InnerVolumeSpecName "kube-api-access-68875". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:39.979006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:39.978982    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbc5b46-2769-4f39-8ef8-b370eef7dfd4-kube-api-access-f2672" pod "4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" (UID: "4cbc5b46-2769-4f39-8ef8-b370eef7dfd4"). InnerVolumeSpecName "kube-api-access-f2672". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:40.080485 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.080443    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2672\" (UniqueName: \"kubernetes.io/projected/4cbc5b46-2769-4f39-8ef8-b370eef7dfd4-kube-api-access-f2672\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:40.080776 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.080738    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68875\" (UniqueName: \"kubernetes.io/projected/9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9-kube-api-access-68875\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:40.112299 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.112202    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:40.116847 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.116813    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" path="/var/lib/kubelet/pods/4cbc5b46-2769-4f39-8ef8-b370eef7dfd4/volumes"
Nov 03 20:52:40.117347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.117322    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" path="/var/lib/kubelet/pods/9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9/volumes"
Nov 03 20:52:40.439521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.439435    5196 generic.go:356] "Generic (PLEG): container finished" podID="9a67d6a6-f4a3-4cd6-9c30-6d21f62e44b9" containerID="1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37" exitCode=137
Nov 03 20:52:40.439521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.439484    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7295-ea7c3e"
Nov 03 20:52:40.440025 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.439550    5196 scope.go:122] "RemoveContainer" containerID="1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37"
Nov 03 20:52:40.442255 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.442217    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q" event={"ID":"e198cf11-c062-47a5-a83f-b109821d5b58","Type":"ContainerStarted","Data":"bac6d06991a5b17ef701abee49e051381363332c83facf713f1ea21f16a47985"}
Nov 03 20:52:40.442396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.442287    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q" event={"ID":"e198cf11-c062-47a5-a83f-b109821d5b58","Type":"ContainerStarted","Data":"e195aa0a297b769a536d9aee8600afcc90ad2c802342cfad9e6a8588ace4d214"}
Nov 03 20:52:40.444282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.444261    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"
Nov 03 20:52:40.444370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.444282    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88" event={"ID":"2fe9721b-7852-454a-869a-94aabb82a390","Type":"ContainerDied","Data":"d6cb72d75e8e29ce277ec750c71bf756b010a5a35de89d9941285b3de5edb2de"}
Nov 03 20:52:40.444370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.444332    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6cb72d75e8e29ce277ec750c71bf756b010a5a35de89d9941285b3de5edb2de"
Nov 03 20:52:40.445849 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.445830    5196 generic.go:356] "Generic (PLEG): container finished" podID="4cbc5b46-2769-4f39-8ef8-b370eef7dfd4" containerID="720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d" exitCode=137
Nov 03 20:52:40.446362 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.445876    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-2c55a4"
Nov 03 20:52:40.452566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.452541    5196 scope.go:122] "RemoveContainer" containerID="1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37"
Nov 03 20:52:40.452843 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:40.452826    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37\": container with ID starting with 1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37 not found: ID does not exist" containerID="1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37"
Nov 03 20:52:40.452896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.452852    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37"} err="failed to get container status \"1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37\": rpc error: code = NotFound desc = could not find container \"1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37\": container with ID starting with 1df2250e1816dc9164e2d2b178a81648c2278e9898cbd70d344855b38e47ce37 not found: ID does not exist"
Nov 03 20:52:40.452896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.452868    5196 scope.go:122] "RemoveContainer" containerID="720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d"
Nov 03 20:52:40.464778 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.464753    5196 scope.go:122] "RemoveContainer" containerID="720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d"
Nov 03 20:52:40.465175 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:40.465150    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d\": container with ID starting with 720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d not found: ID does not exist" containerID="720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d"
Nov 03 20:52:40.465248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.465188    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d"} err="failed to get container status \"720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d\": rpc error: code = NotFound desc = could not find container \"720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d\": container with ID starting with 720d9d376022ceaeb0138440900ecc1abaf2b7ec78a504688b345e9cf6d12f2d not found: ID does not exist"
Nov 03 20:52:40.554996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:40.554938    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q" podStartSLOduration=2.554923393 podStartE2EDuration="2.554923393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:40.553825591 +0000 UTC m=+1400.557784334" watchObservedRunningTime="2026-11-03 20:52:40.554923393 +0000 UTC m=+1400.558882167"
Nov 03 20:52:41.459774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:41.459742    5196 generic.go:356] "Generic (PLEG): container finished" podID="e198cf11-c062-47a5-a83f-b109821d5b58" containerID="bac6d06991a5b17ef701abee49e051381363332c83facf713f1ea21f16a47985" exitCode=0
Nov 03 20:52:41.460301 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:41.459799    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q" event={"ID":"e198cf11-c062-47a5-a83f-b109821d5b58","Type":"ContainerDied","Data":"bac6d06991a5b17ef701abee49e051381363332c83facf713f1ea21f16a47985"}
Nov 03 20:52:41.469216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:41.469178    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:41.913020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:41.912988    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:52:42.897789 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:42.897753    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:42.971929 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:42.970922    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-xks-ca\") pod \"e198cf11-c062-47a5-a83f-b109821d5b58\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") "
Nov 03 20:52:42.971929 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:42.971012    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-openshift-ca\") pod \"e198cf11-c062-47a5-a83f-b109821d5b58\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") "
Nov 03 20:52:42.972686 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:42.972375    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-openshift-ca" pod "e198cf11-c062-47a5-a83f-b109821d5b58" (UID: "e198cf11-c062-47a5-a83f-b109821d5b58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:43.003131 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.000329    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-xks-ca" pod "e198cf11-c062-47a5-a83f-b109821d5b58" (UID: "e198cf11-c062-47a5-a83f-b109821d5b58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:43.126121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.110736    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e198cf11-c062-47a5-a83f-b109821d5b58-kube-api-access-nxxrn\" (UniqueName: \"kubernetes.io/projected/e198cf11-c062-47a5-a83f-b109821d5b58-kube-api-access-nxxrn\") pod \"e198cf11-c062-47a5-a83f-b109821d5b58\" (UID: \"e198cf11-c062-47a5-a83f-b109821d5b58\") "
Nov 03 20:52:43.155120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.154671    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:43.155120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.154711    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e198cf11-c062-47a5-a83f-b109821d5b58-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:43.434191 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.431723    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"]
Nov 03 20:52:43.601781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.598223    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449d4ab1-rjwzx"]
Nov 03 20:52:43.620928 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.618139    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e198cf11-c062-47a5-a83f-b109821d5b58-kube-api-access-nxxrn" pod "e198cf11-c062-47a5-a83f-b109821d5b58" (UID: "e198cf11-c062-47a5-a83f-b109821d5b58"). InnerVolumeSpecName "kube-api-access-nxxrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:43.666515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.640813    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxrn\" (UniqueName: \"kubernetes.io/projected/e198cf11-c062-47a5-a83f-b109821d5b58-kube-api-access-nxxrn\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:43.666515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.655212    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:43.763706 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.763321    5196 generic.go:356] "Generic (PLEG): container finished" podID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerID="6742fc52080a7f35a474eaea9e7e4fceb139c500feba733865fd1516cdbef08f" exitCode=0
Nov 03 20:52:43.763706 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.763613    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" event={"ID":"463aab31-1e86-464e-97c4-d85d3a7d4f2a","Type":"ContainerDied","Data":"6742fc52080a7f35a474eaea9e7e4fceb139c500feba733865fd1516cdbef08f"}
Nov 03 20:52:43.787731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.781988    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q" event={"ID":"e198cf11-c062-47a5-a83f-b109821d5b58","Type":"ContainerDied","Data":"e195aa0a297b769a536d9aee8600afcc90ad2c802342cfad9e6a8588ace4d214"}
Nov 03 20:52:43.787731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.782058    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e195aa0a297b769a536d9aee8600afcc90ad2c802342cfad9e6a8588ace4d214"
Nov 03 20:52:43.787731 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:43.782163    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"
Nov 03 20:52:44.170122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.154432    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:44.227704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.227664    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a09464-8c6a-4d5d-802e-a7a87e38f904" path="/var/lib/kubelet/pods/66a09464-8c6a-4d5d-802e-a7a87e38f904/volumes"
Nov 03 20:52:44.308242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.308199    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:44.322711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.322676    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:44.517591 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.517481    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 20:52:44.700129 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.699231    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:44.912118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.911707    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:44.936123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.933216    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 20:52:44.972122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.971943    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": context deadline exceeded"
Nov 03 20:52:44.972122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:44.972022    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:45.041120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.040583    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:45.053385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.041438    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:45.199184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.198265    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:45.318126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.316074    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:45.318126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.316303    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:52:45.520548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.520244    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:45.559125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.520675    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/health\": context deadline exceeded"
Nov 03 20:52:45.559125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.521143    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:52:45.738121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.703500    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": context deadline exceeded"
Nov 03 20:52:46.016125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.973306    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.016125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.919228    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.016125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.974484    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:52:46.016125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.974550    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:52:46.016125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.974591    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:52:46.016125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:45.937473    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.016125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.002244    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:52:46.050323 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.050271    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 03 20:52:46.151123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.051465    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": context deadline exceeded"
Nov 03 20:52:46.211982 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.116009    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:52:46.229451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.229415    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:52:46.467373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.198433    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.467373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.296944    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:52:46.467373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.351826    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.467373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.446764    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-btmgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:46.467373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.446820    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.588788 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.531924    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.588788 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.544893    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"
Nov 03 20:52:46.617412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.617361    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.678130 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.674821    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:52:46.787185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.787066    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/health\": context deadline exceeded"
Nov 03 20:52:46.805121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.798333    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:46.888146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.686207    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg" event={"ID":"463aab31-1e86-464e-97c4-d85d3a7d4f2a","Type":"ContainerDied","Data":"9156cc8f3c2e0485a1e6d2a8419d0ad7fe3a0a618d751fe2dbd5818d67f736fd"}
Nov 03 20:52:47.043047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.042955    5196 scope.go:122] "RemoveContainer" containerID="6742fc52080a7f35a474eaea9e7e4fceb139c500feba733865fd1516cdbef08f"
Nov 03 20:52:47.069155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.899181    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-certs\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.069894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.069854    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-socket\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.205727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.135123    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-kube-api-access-b5hsw\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-kube-api-access-b5hsw\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.205727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.135296    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istiod-ca-cert\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.205727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.135378    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-token\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.205727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.135424    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-credential-socket\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.205727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.135509    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-envoy\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.205727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.135538    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-data\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.205727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.135638    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-podinfo\") pod \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\" (UID: \"463aab31-1e86-464e-97c4-d85d3a7d4f2a\") "
Nov 03 20:52:47.205727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.961657    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.220165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.208285    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.320183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.979415    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.369223 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.269899    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.401165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.979454    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.430171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.426474    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.626170 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.979475    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.667126 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.979504    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.692124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.680501    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.760162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:46.920238    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-certs" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:47.760379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.220674    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-socket" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:47.760497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.370341    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-kube-api-access-b5hsw" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "kube-api-access-b5hsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:47.760587 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.625414    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istiod-ca-cert" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:47.760674 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.433981    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.798122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.762118    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-token" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:47.865121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.864267    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-credential-socket" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:47.874819 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.874413    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-data" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:47.926385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.924655    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:47.926385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.924768    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.978641 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:47.978588    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-podinfo" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:52:48.016375 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.016332    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-envoy" pod "463aab31-1e86-464e-97c4-d85d3a7d4f2a" (UID: "463aab31-1e86-464e-97c4-d85d3a7d4f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:48.223330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.223216    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.278436 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.277217    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:48.290794 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.290762    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:52:48.315589 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.315532    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.345414 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.345359    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.400313 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.400248    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.534660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.465269    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.534660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.471210    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.534660 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.507211    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:48.582431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.519084    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.584538 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.584503    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.597599 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.597564    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.597872 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.597851    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.598057 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.598040    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.598201 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.598185    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hsw\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-kube-api-access-b5hsw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.598375 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.598357    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.623786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.623746    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463aab31-1e86-464e-97c4-d85d3a7d4f2a-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.624359 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.624330    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463aab31-1e86-464e-97c4-d85d3a7d4f2a-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:48.638750 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.638701    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.674121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.648773    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.878990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.771436    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.879379 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.790066    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:48.975630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.969677    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.975630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.801237    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.975630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.801194    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.975630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.878649    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:48.975630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.951232    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7nr6l" podUID="822db250-6afd-4838-b96d-b64f1b7dbfc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.977779 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:48.977740    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g" podUID="e43f5087-4ed9-4e95-bf72-57432ed7681c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.79:50052: i/o timeout"
Nov 03 20:52:49.323186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.307197    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.401239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.350656    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.401239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.351073    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.467272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.467223    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.477266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.477220    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.663185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.653581    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:49.667308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.667261    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.899121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.898235    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.920174 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:49.912954    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.009291 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.009250    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.025661 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.025608    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.054133 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.040601    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 20:52:50.054133 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.053593    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Nov 03 20:52:50.128124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.128055    5196 patch_prober.go:29] interesting pod/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:50.128421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.128385    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.304294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.304200    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:50.304545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.304511    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.310173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.310135    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.329118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.322587    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:50.329729 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.329693    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.403659 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.403460    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 20:52:50.408745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.404168    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 03 20:52:50.433961 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:50.433923    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:52:50.438393 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.438343    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:52:50.489277 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.487557    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.514332 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.514292    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.516320 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.516295    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:52:50.612267 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.611492    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:52:50.631127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.629425    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:52:50.633373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.632501    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:52:50.657131 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.656765    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:52:50.727045 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.721177    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:52:50.736065 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:52:50.736028    5196 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463aab31_1e86_464e_97c4_d85d3a7d4f2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463aab31_1e86_464e_97c4_d85d3a7d4f2a.slice/crio-9156cc8f3c2e0485a1e6d2a8419d0ad7fe3a0a618d751fe2dbd5818d67f736fd\": RecentStats: unable to find data in memory cache]"
Nov 03 20:52:50.754698 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.754051    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"]
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773017    5196 cpu_manager.go:414] "containerMap: removing container" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy"
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773050    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy"
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773087    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e198cf11-c062-47a5-a83f-b109821d5b58" containerName="revoke-keys"
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773115    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e198cf11-c062-47a5-a83f-b109821d5b58" containerName="revoke-keys"
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773184    5196 cpu_manager.go:414] "containerMap: removing container" podUID="2fe9721b-7852-454a-869a-94aabb82a390" containerName="revoke-keys"
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773191    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe9721b-7852-454a-869a-94aabb82a390" containerName="revoke-keys"
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773591    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe9721b-7852-454a-869a-94aabb82a390" containerName="revoke-keys"
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773641    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" containerName="istio-proxy"
Nov 03 20:52:50.777522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.773653    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e198cf11-c062-47a5-a83f-b109821d5b58" containerName="revoke-keys"
Nov 03 20:52:50.789159 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.789125    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:52:50.789353 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.789179    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"]
Nov 03 20:52:50.789353 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.789217    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"]
Nov 03 20:52:50.791061 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.790948    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:50.800386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.800366    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:52:50.800853 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.800825    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:52:50.801023 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.800593    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:50.801329 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.801193    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"]
Nov 03 20:52:50.843610 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.843580    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:52:50.843610 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.843584    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:52:50.913197 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.913069    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rbq\" (UniqueName: \"kubernetes.io/projected/8150b373-242f-4dde-a30b-293d48b2b3e6-kube-api-access-75rbq\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:50.913439 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.913416    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vzf\" (UniqueName: \"kubernetes.io/projected/295a2c23-a8e9-4929-8716-a99c15ac52b7-kube-api-access-74vzf\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:50.913566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.913542    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:50.913712 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.913690    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:50.913814 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.913776    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:50.913921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:50.913872    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:51.015743 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.015710    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vzf\" (UniqueName: \"kubernetes.io/projected/295a2c23-a8e9-4929-8716-a99c15ac52b7-kube-api-access-74vzf\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:51.015912 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.015785    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:51.015912 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.015858    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:51.015912 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.015902    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:51.016088 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.015920    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:51.016088 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.016071    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rbq\" (UniqueName: \"kubernetes.io/projected/8150b373-242f-4dde-a30b-293d48b2b3e6-kube-api-access-75rbq\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:51.016799 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.016768    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:51.016978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.016852    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:51.018716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.018695    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:51.018814 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.018786    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:51.075057 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.075022    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vzf\" (UniqueName: \"kubernetes.io/projected/295a2c23-a8e9-4929-8716-a99c15ac52b7-kube-api-access-74vzf\") pod \"maas-api-revoke-sub-ddc12832-sdj8p\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:51.075298 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.075272    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rbq\" (UniqueName: \"kubernetes.io/projected/8150b373-242f-4dde-a30b-293d48b2b3e6-kube-api-access-75rbq\") pod \"maas-api-revoke-sub-83dd62e7-8f6ql\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:51.119604 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.119563    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:51.129951 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.129924    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:51.207936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.206553    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:51.536161 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.535884    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"]
Nov 03 20:52:51.536161 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:51.536002    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295a2c23_a8e9_4929_8716_a99c15ac52b7.slice/crio-a72eca5abaaeaff43f13141be31537e5a3b4acdbc4b4d70c6cb748eb42395018 WatchSource:0}: Error finding container a72eca5abaaeaff43f13141be31537e5a3b4acdbc4b4d70c6cb748eb42395018: Status 404 returned error can't find the container with id a72eca5abaaeaff43f13141be31537e5a3b4acdbc4b4d70c6cb748eb42395018
Nov 03 20:52:51.586564 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.586531    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"]
Nov 03 20:52:51.591897 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:51.591861    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8150b373_242f_4dde_a30b_293d48b2b3e6.slice/crio-40c8d29628696e13d20c13ecfd14068159917866597f3a79d4b4e8274098c6f5 WatchSource:0}: Error finding container 40c8d29628696e13d20c13ecfd14068159917866597f3a79d4b4e8274098c6f5: Status 404 returned error can't find the container with id 40c8d29628696e13d20c13ecfd14068159917866597f3a79d4b4e8274098c6f5
Nov 03 20:52:51.608529 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.608498    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"]
Nov 03 20:52:51.614366 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.614330    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-btmgg"]
Nov 03 20:52:51.619847 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.619815    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"]
Nov 03 20:52:51.626399 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.626374    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.695011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.694953    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"]
Nov 03 20:52:51.728196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.728157    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2crx\" (UniqueName: \"kubernetes.io/projected/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-kube-api-access-g2crx\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.728402 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.728386    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.728540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.728518    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.829890 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.829855    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.830117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.829982    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.830117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.830053    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2crx\" (UniqueName: \"kubernetes.io/projected/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-kube-api-access-g2crx\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.830680 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.830654    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.833342 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.833310    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.884309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.884280    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2crx\" (UniqueName: \"kubernetes.io/projected/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-kube-api-access-g2crx\") pod \"maas-api-revoke-sub-6cf3ec3a-hp644\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:51.958007 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:51.957974    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:52.115639 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.115552    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463aab31-1e86-464e-97c4-d85d3a7d4f2a" path="/var/lib/kubelet/pods/463aab31-1e86-464e-97c4-d85d3a7d4f2a/volumes"
Nov 03 20:52:52.368242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.368154    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"]
Nov 03 20:52:52.372255 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:52.372225    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c7b3f5_671f_49c3_a318_b2ee0f23e0a7.slice/crio-65f4509063be17be0bcc33d0ba86a321634de24da35c6463ac395aa338ec0ff9 WatchSource:0}: Error finding container 65f4509063be17be0bcc33d0ba86a321634de24da35c6463ac395aa338ec0ff9: Status 404 returned error can't find the container with id 65f4509063be17be0bcc33d0ba86a321634de24da35c6463ac395aa338ec0ff9
Nov 03 20:52:52.377390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.377356    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p" event={"ID":"295a2c23-a8e9-4929-8716-a99c15ac52b7","Type":"ContainerStarted","Data":"7b948272ad3f4dc1fe42365c1872a49abdfee3757ae0882703c909b75be62bed"}
Nov 03 20:52:52.377519 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.377418    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p" event={"ID":"295a2c23-a8e9-4929-8716-a99c15ac52b7","Type":"ContainerStarted","Data":"a72eca5abaaeaff43f13141be31537e5a3b4acdbc4b4d70c6cb748eb42395018"}
Nov 03 20:52:52.379679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.379652    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql" event={"ID":"8150b373-242f-4dde-a30b-293d48b2b3e6","Type":"ContainerStarted","Data":"6107d14f5a30d79bd17771b2d22f41be1353d7e94186508316d49c3a47865e60"}
Nov 03 20:52:52.379782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.379709    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql" event={"ID":"8150b373-242f-4dde-a30b-293d48b2b3e6","Type":"ContainerStarted","Data":"40c8d29628696e13d20c13ecfd14068159917866597f3a79d4b4e8274098c6f5"}
Nov 03 20:52:52.486695 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.486637    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p" podStartSLOduration=3.486622441 podStartE2EDuration="3.486622441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:52.484007781 +0000 UTC m=+1412.487966522" watchObservedRunningTime="2026-11-03 20:52:52.486622441 +0000 UTC m=+1412.490581165"
Nov 03 20:52:52.605240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.605183    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql" podStartSLOduration=2.60516308 podStartE2EDuration="2.60516308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:52.59966531 +0000 UTC m=+1412.603624072" watchObservedRunningTime="2026-11-03 20:52:52.60516308 +0000 UTC m=+1412.609121804"
Nov 03 20:52:52.884421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.884385    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"]
Nov 03 20:52:52.889685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.889658    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:52.950643 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:52.950549    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"]
Nov 03 20:52:53.051722 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.051685    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.051915 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.051737    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.051915 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.051829    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6wz\" (UniqueName: \"kubernetes.io/projected/4ca11177-4162-4f77-8a25-fc2b4fd584f1-kube-api-access-rk6wz\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.153360 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.153320    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.153558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.153379    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.153558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.153417    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6wz\" (UniqueName: \"kubernetes.io/projected/4ca11177-4162-4f77-8a25-fc2b4fd584f1-kube-api-access-rk6wz\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.154057 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.154035    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.156122 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.156078    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.207185 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.207118    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6wz\" (UniqueName: \"kubernetes.io/projected/4ca11177-4162-4f77-8a25-fc2b4fd584f1-kube-api-access-rk6wz\") pod \"maas-api-revoke-sub-c5925af8-g5lqr\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.391746 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.391707    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644" event={"ID":"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7","Type":"ContainerStarted","Data":"e4128f0d4576b2cc59140b3947754682237a871c0661bf61c74e1fa069482cfe"}
Nov 03 20:52:53.391977 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.391767    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644" event={"ID":"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7","Type":"ContainerStarted","Data":"65f4509063be17be0bcc33d0ba86a321634de24da35c6463ac395aa338ec0ff9"}
Nov 03 20:52:53.394413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.394378    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p" event={"ID":"295a2c23-a8e9-4929-8716-a99c15ac52b7","Type":"ContainerDied","Data":"7b948272ad3f4dc1fe42365c1872a49abdfee3757ae0882703c909b75be62bed"}
Nov 03 20:52:53.394413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.394301    5196 generic.go:356] "Generic (PLEG): container finished" podID="295a2c23-a8e9-4929-8716-a99c15ac52b7" containerID="7b948272ad3f4dc1fe42365c1872a49abdfee3757ae0882703c909b75be62bed" exitCode=0
Nov 03 20:52:53.396634 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.396608    5196 generic.go:356] "Generic (PLEG): container finished" podID="8150b373-242f-4dde-a30b-293d48b2b3e6" containerID="6107d14f5a30d79bd17771b2d22f41be1353d7e94186508316d49c3a47865e60" exitCode=0
Nov 03 20:52:53.396775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.396632    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql" event={"ID":"8150b373-242f-4dde-a30b-293d48b2b3e6","Type":"ContainerDied","Data":"6107d14f5a30d79bd17771b2d22f41be1353d7e94186508316d49c3a47865e60"}
Nov 03 20:52:53.499438 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.499337    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644" podStartSLOduration=2.4993235560000002 podStartE2EDuration="2.499323556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:53.498300747 +0000 UTC m=+1413.502259493" watchObservedRunningTime="2026-11-03 20:52:53.499323556 +0000 UTC m=+1413.503282280"
Nov 03 20:52:53.506918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.506890    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:53.899970 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:53.899938    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"]
Nov 03 20:52:53.903318 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:53.903291    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca11177_4162_4f77_8a25_fc2b4fd584f1.slice/crio-e7da8c7cb40b6ffa197a9c18cb0074fe298bc75d9e8fd738dda3032cc1695a1f WatchSource:0}: Error finding container e7da8c7cb40b6ffa197a9c18cb0074fe298bc75d9e8fd738dda3032cc1695a1f: Status 404 returned error can't find the container with id e7da8c7cb40b6ffa197a9c18cb0074fe298bc75d9e8fd738dda3032cc1695a1f
Nov 03 20:52:54.408766 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.408674    5196 generic.go:356] "Generic (PLEG): container finished" podID="30c7b3f5-671f-49c3-a318-b2ee0f23e0a7" containerID="e4128f0d4576b2cc59140b3947754682237a871c0661bf61c74e1fa069482cfe" exitCode=0
Nov 03 20:52:54.408766 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.408734    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644" event={"ID":"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7","Type":"ContainerDied","Data":"e4128f0d4576b2cc59140b3947754682237a871c0661bf61c74e1fa069482cfe"}
Nov 03 20:52:54.410999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.410975    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr" event={"ID":"4ca11177-4162-4f77-8a25-fc2b4fd584f1","Type":"ContainerStarted","Data":"a7130a0f286e06d80c0d2b20de6aff1703141607952b72a97df4d6e86aa8e308"}
Nov 03 20:52:54.411184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.411018    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr" event={"ID":"4ca11177-4162-4f77-8a25-fc2b4fd584f1","Type":"ContainerStarted","Data":"e7da8c7cb40b6ffa197a9c18cb0074fe298bc75d9e8fd738dda3032cc1695a1f"}
Nov 03 20:52:54.420196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.420163    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:54.579588 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.579557    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:54.623236 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.623145    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:54.632729 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.632673    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr" podStartSLOduration=2.6326558909999997 podStartE2EDuration="2.632655891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:54.625650683 +0000 UTC m=+1414.629609427" watchObservedRunningTime="2026-11-03 20:52:54.632655891 +0000 UTC m=+1414.636614611"
Nov 03 20:52:54.664759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.664677    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:54.675280 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.675244    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-openshift-ca\") pod \"8150b373-242f-4dde-a30b-293d48b2b3e6\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") "
Nov 03 20:52:54.675458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.675342    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-xks-ca\") pod \"295a2c23-a8e9-4929-8716-a99c15ac52b7\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") "
Nov 03 20:52:54.675458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.675443    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-openshift-ca\") pod \"295a2c23-a8e9-4929-8716-a99c15ac52b7\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") "
Nov 03 20:52:54.675573 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.675482    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-xks-ca\") pod \"8150b373-242f-4dde-a30b-293d48b2b3e6\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") "
Nov 03 20:52:54.675573 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.675511    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8150b373-242f-4dde-a30b-293d48b2b3e6-kube-api-access-75rbq\" (UniqueName: \"kubernetes.io/projected/8150b373-242f-4dde-a30b-293d48b2b3e6-kube-api-access-75rbq\") pod \"8150b373-242f-4dde-a30b-293d48b2b3e6\" (UID: \"8150b373-242f-4dde-a30b-293d48b2b3e6\") "
Nov 03 20:52:54.675685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.675571    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295a2c23-a8e9-4929-8716-a99c15ac52b7-kube-api-access-74vzf\" (UniqueName: \"kubernetes.io/projected/295a2c23-a8e9-4929-8716-a99c15ac52b7-kube-api-access-74vzf\") pod \"295a2c23-a8e9-4929-8716-a99c15ac52b7\" (UID: \"295a2c23-a8e9-4929-8716-a99c15ac52b7\") "
Nov 03 20:52:54.675685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.675594    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-openshift-ca" pod "8150b373-242f-4dde-a30b-293d48b2b3e6" (UID: "8150b373-242f-4dde-a30b-293d48b2b3e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:54.675852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.675830    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-openshift-ca" pod "295a2c23-a8e9-4929-8716-a99c15ac52b7" (UID: "295a2c23-a8e9-4929-8716-a99c15ac52b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:54.676285 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.676231    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:54.676285 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.676254    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:54.677836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.677814    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-xks-ca" pod "8150b373-242f-4dde-a30b-293d48b2b3e6" (UID: "8150b373-242f-4dde-a30b-293d48b2b3e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:54.677942 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.677862    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-xks-ca" pod "295a2c23-a8e9-4929-8716-a99c15ac52b7" (UID: "295a2c23-a8e9-4929-8716-a99c15ac52b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:54.678357 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.678335    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295a2c23-a8e9-4929-8716-a99c15ac52b7-kube-api-access-74vzf" pod "295a2c23-a8e9-4929-8716-a99c15ac52b7" (UID: "295a2c23-a8e9-4929-8716-a99c15ac52b7"). InnerVolumeSpecName "kube-api-access-74vzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:54.678446 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.678364    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8150b373-242f-4dde-a30b-293d48b2b3e6-kube-api-access-75rbq" pod "8150b373-242f-4dde-a30b-293d48b2b3e6" (UID: "8150b373-242f-4dde-a30b-293d48b2b3e6"). InnerVolumeSpecName "kube-api-access-75rbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:54.748818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.748763    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:54.779161 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.779124    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vzf\" (UniqueName: \"kubernetes.io/projected/295a2c23-a8e9-4929-8716-a99c15ac52b7-kube-api-access-74vzf\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:54.779161 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.779162    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295a2c23-a8e9-4929-8716-a99c15ac52b7-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:54.779369 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.779177    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8150b373-242f-4dde-a30b-293d48b2b3e6-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:54.779369 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:54.779191    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rbq\" (UniqueName: \"kubernetes.io/projected/8150b373-242f-4dde-a30b-293d48b2b3e6-kube-api-access-75rbq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:55.428194 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.428166    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"
Nov 03 20:52:55.429787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.428158    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql" event={"ID":"8150b373-242f-4dde-a30b-293d48b2b3e6","Type":"ContainerDied","Data":"40c8d29628696e13d20c13ecfd14068159917866597f3a79d4b4e8274098c6f5"}
Nov 03 20:52:55.429787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.429748    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40c8d29628696e13d20c13ecfd14068159917866597f3a79d4b4e8274098c6f5"
Nov 03 20:52:55.430872 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.430849    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"
Nov 03 20:52:55.430992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.430866    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p" event={"ID":"295a2c23-a8e9-4929-8716-a99c15ac52b7","Type":"ContainerDied","Data":"a72eca5abaaeaff43f13141be31537e5a3b4acdbc4b4d70c6cb748eb42395018"}
Nov 03 20:52:55.430992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.430911    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a72eca5abaaeaff43f13141be31537e5a3b4acdbc4b4d70c6cb748eb42395018"
Nov 03 20:52:55.433958 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.433647    5196 generic.go:356] "Generic (PLEG): container finished" podID="4ca11177-4162-4f77-8a25-fc2b4fd584f1" containerID="a7130a0f286e06d80c0d2b20de6aff1703141607952b72a97df4d6e86aa8e308" exitCode=0
Nov 03 20:52:55.434409 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.434380    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr" event={"ID":"4ca11177-4162-4f77-8a25-fc2b4fd584f1","Type":"ContainerDied","Data":"a7130a0f286e06d80c0d2b20de6aff1703141607952b72a97df4d6e86aa8e308"}
Nov 03 20:52:55.509245 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.508723    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:55.715667 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.715590    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:55.810239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.810204    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-openshift-ca\") pod \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") "
Nov 03 20:52:55.810454 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.810432    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-xks-ca\") pod \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") "
Nov 03 20:52:55.810454 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.810454    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-kube-api-access-g2crx\" (UniqueName: \"kubernetes.io/projected/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-kube-api-access-g2crx\") pod \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\" (UID: \"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7\") "
Nov 03 20:52:55.810670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.810645    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-openshift-ca" pod "30c7b3f5-671f-49c3-a318-b2ee0f23e0a7" (UID: "30c7b3f5-671f-49c3-a318-b2ee0f23e0a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:55.811034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.811013    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:55.812883 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.812855    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-xks-ca" pod "30c7b3f5-671f-49c3-a318-b2ee0f23e0a7" (UID: "30c7b3f5-671f-49c3-a318-b2ee0f23e0a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:55.813363 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.813343    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-kube-api-access-g2crx" pod "30c7b3f5-671f-49c3-a318-b2ee0f23e0a7" (UID: "30c7b3f5-671f-49c3-a318-b2ee0f23e0a7"). InnerVolumeSpecName "kube-api-access-g2crx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:55.904710 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.904674    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:55.912913 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.912837    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:55.912913 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:55.912870    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2crx\" (UniqueName: \"kubernetes.io/projected/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7-kube-api-access-g2crx\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.280049 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.280014    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:56.448572 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.448476    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"
Nov 03 20:52:56.448572 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.448475    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644" event={"ID":"30c7b3f5-671f-49c3-a318-b2ee0f23e0a7","Type":"ContainerDied","Data":"65f4509063be17be0bcc33d0ba86a321634de24da35c6463ac395aa338ec0ff9"}
Nov 03 20:52:56.448572 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.448568    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65f4509063be17be0bcc33d0ba86a321634de24da35c6463ac395aa338ec0ff9"
Nov 03 20:52:56.658390 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.658358    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:56.734146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.734046    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca11177-4162-4f77-8a25-fc2b4fd584f1-kube-api-access-rk6wz\" (UniqueName: \"kubernetes.io/projected/4ca11177-4162-4f77-8a25-fc2b4fd584f1-kube-api-access-rk6wz\") pod \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") "
Nov 03 20:52:56.734299 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.734263    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-xks-ca\") pod \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") "
Nov 03 20:52:56.734427 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.734407    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-openshift-ca\") pod \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\" (UID: \"4ca11177-4162-4f77-8a25-fc2b4fd584f1\") "
Nov 03 20:52:56.734910 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.734878    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-openshift-ca" pod "4ca11177-4162-4f77-8a25-fc2b4fd584f1" (UID: "4ca11177-4162-4f77-8a25-fc2b4fd584f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:56.735289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.735268    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.736676 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.736652    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-xks-ca" pod "4ca11177-4162-4f77-8a25-fc2b4fd584f1" (UID: "4ca11177-4162-4f77-8a25-fc2b4fd584f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:56.736773 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.736753    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca11177-4162-4f77-8a25-fc2b4fd584f1-kube-api-access-rk6wz" pod "4ca11177-4162-4f77-8a25-fc2b4fd584f1" (UID: "4ca11177-4162-4f77-8a25-fc2b4fd584f1"). InnerVolumeSpecName "kube-api-access-rk6wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:56.836395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.836359    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6wz\" (UniqueName: \"kubernetes.io/projected/4ca11177-4162-4f77-8a25-fc2b4fd584f1-kube-api-access-rk6wz\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.836395 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:56.836388    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca11177-4162-4f77-8a25-fc2b4fd584f1-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:57.037868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.037775    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:57.229676 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.229627    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:57.434621 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.434583    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"]
Nov 03 20:52:57.435556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435531    5196 cpu_manager.go:414] "containerMap: removing container" podUID="30c7b3f5-671f-49c3-a318-b2ee0f23e0a7" containerName="revoke-keys"
Nov 03 20:52:57.435556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435554    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c7b3f5-671f-49c3-a318-b2ee0f23e0a7" containerName="revoke-keys"
Nov 03 20:52:57.435756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435578    5196 cpu_manager.go:414] "containerMap: removing container" podUID="295a2c23-a8e9-4929-8716-a99c15ac52b7" containerName="revoke-keys"
Nov 03 20:52:57.435756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435585    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295a2c23-a8e9-4929-8716-a99c15ac52b7" containerName="revoke-keys"
Nov 03 20:52:57.435756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435599    5196 cpu_manager.go:414] "containerMap: removing container" podUID="8150b373-242f-4dde-a30b-293d48b2b3e6" containerName="revoke-keys"
Nov 03 20:52:57.435756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435606    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8150b373-242f-4dde-a30b-293d48b2b3e6" containerName="revoke-keys"
Nov 03 20:52:57.435756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435636    5196 cpu_manager.go:414] "containerMap: removing container" podUID="4ca11177-4162-4f77-8a25-fc2b4fd584f1" containerName="revoke-keys"
Nov 03 20:52:57.435756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435643    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca11177-4162-4f77-8a25-fc2b4fd584f1" containerName="revoke-keys"
Nov 03 20:52:57.436046 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435857    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c7b3f5-671f-49c3-a318-b2ee0f23e0a7" containerName="revoke-keys"
Nov 03 20:52:57.436046 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435871    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="295a2c23-a8e9-4929-8716-a99c15ac52b7" containerName="revoke-keys"
Nov 03 20:52:57.436046 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435883    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="8150b373-242f-4dde-a30b-293d48b2b3e6" containerName="revoke-keys"
Nov 03 20:52:57.436046 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.435897    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca11177-4162-4f77-8a25-fc2b4fd584f1" containerName="revoke-keys"
Nov 03 20:52:57.445246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.440484    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:57.467146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.467111    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"
Nov 03 20:52:57.467647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.467201    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr" event={"ID":"4ca11177-4162-4f77-8a25-fc2b4fd584f1","Type":"ContainerDied","Data":"e7da8c7cb40b6ffa197a9c18cb0074fe298bc75d9e8fd738dda3032cc1695a1f"}
Nov 03 20:52:57.467647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.467277    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7da8c7cb40b6ffa197a9c18cb0074fe298bc75d9e8fd738dda3032cc1695a1f"
Nov 03 20:52:57.489677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.489642    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e20715ef-dockercfg-rvrrm\""
Nov 03 20:52:57.489677 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.489644    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e20715ef\""
Nov 03 20:52:57.491729 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.491692    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"]
Nov 03 20:52:57.547504 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.547468    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhc4d\" (UniqueName: \"kubernetes.io/projected/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-kube-api-access-hhc4d\") pod \"payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd\" (UID: \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:57.547689 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.547622    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd\" (UID: \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:57.589408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.589368    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"]
Nov 03 20:52:57.594495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.594473    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:57.649158 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.649052    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c0699-760a-4073-99c2-903788909dba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg\" (UID: \"5f2c0699-760a-4073-99c2-903788909dba\") " pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:57.649426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.649399    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"]
Nov 03 20:52:57.649712 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.649685    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhc4d\" (UniqueName: \"kubernetes.io/projected/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-kube-api-access-hhc4d\") pod \"payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd\" (UID: \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:57.649938 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.649911    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd\" (UID: \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:57.687216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.649986    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j24m\" (UniqueName: \"kubernetes.io/projected/5f2c0699-760a-4073-99c2-903788909dba-kube-api-access-6j24m\") pod \"payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg\" (UID: \"5f2c0699-760a-4073-99c2-903788909dba\") " pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:57.687216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.651527    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd\" (UID: \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:57.723038 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.722998    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhc4d\" (UniqueName: \"kubernetes.io/projected/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-kube-api-access-hhc4d\") pod \"payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd\" (UID: \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:57.765394 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.765360    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:57.788138 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.788105    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j24m\" (UniqueName: \"kubernetes.io/projected/5f2c0699-760a-4073-99c2-903788909dba-kube-api-access-6j24m\") pod \"payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg\" (UID: \"5f2c0699-760a-4073-99c2-903788909dba\") " pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:57.788319 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.788181    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c0699-760a-4073-99c2-903788909dba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg\" (UID: \"5f2c0699-760a-4073-99c2-903788909dba\") " pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:57.789018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.788980    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c0699-760a-4073-99c2-903788909dba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg\" (UID: \"5f2c0699-760a-4073-99c2-903788909dba\") " pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:57.844671 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.844643    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j24m\" (UniqueName: \"kubernetes.io/projected/5f2c0699-760a-4073-99c2-903788909dba-kube-api-access-6j24m\") pod \"payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg\" (UID: \"5f2c0699-760a-4073-99c2-903788909dba\") " pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:57.920152 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:57.914543    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:58.186287 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:58.186248    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cbe6b2_ff91_490d_9491_6cdcb8e201e5.slice/crio-b5b2990c26e7f9cacd9e5553430f610b6a6fd975282acb1dc7805ebe8e6fb86f WatchSource:0}: Error finding container b5b2990c26e7f9cacd9e5553430f610b6a6fd975282acb1dc7805ebe8e6fb86f: Status 404 returned error can't find the container with id b5b2990c26e7f9cacd9e5553430f610b6a6fd975282acb1dc7805ebe8e6fb86f
Nov 03 20:52:58.187705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.187678    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"]
Nov 03 20:52:58.331018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.330981    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"]
Nov 03 20:52:58.331185 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:52:58.331079    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2c0699_760a_4073_99c2_903788909dba.slice/crio-8d56cddfd05bb441dbd357392c2e5b4fa9707be534ebe2660dd1ea802c6405c7 WatchSource:0}: Error finding container 8d56cddfd05bb441dbd357392c2e5b4fa9707be534ebe2660dd1ea802c6405c7: Status 404 returned error can't find the container with id 8d56cddfd05bb441dbd357392c2e5b4fa9707be534ebe2660dd1ea802c6405c7
Nov 03 20:52:58.479326 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.479227    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd" event={"ID":"49cbe6b2-ff91-490d-9491-6cdcb8e201e5","Type":"ContainerStarted","Data":"1fec91e4a26d4313a1720a42f8e8d6e229af6ef28320d445563abadfb9e2b9c5"}
Nov 03 20:52:58.479326 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.479299    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd" event={"ID":"49cbe6b2-ff91-490d-9491-6cdcb8e201e5","Type":"ContainerStarted","Data":"b5b2990c26e7f9cacd9e5553430f610b6a6fd975282acb1dc7805ebe8e6fb86f"}
Nov 03 20:52:58.479788 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.479330    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:52:58.481575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.481525    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg" event={"ID":"5f2c0699-760a-4073-99c2-903788909dba","Type":"ContainerStarted","Data":"d44bc83ac964f190efa83c2ef27fb6de5f85a8d9dcf7c097c415981c906522cb"}
Nov 03 20:52:58.481714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.481587    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg" event={"ID":"5f2c0699-760a-4073-99c2-903788909dba","Type":"ContainerStarted","Data":"8d56cddfd05bb441dbd357392c2e5b4fa9707be534ebe2660dd1ea802c6405c7"}
Nov 03 20:52:58.481714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.481610    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:52:58.588963 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.588901    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd" podStartSLOduration=1.5888828720000001 podStartE2EDuration="1.588882872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:58.587247245 +0000 UTC m=+1418.591205989" watchObservedRunningTime="2026-11-03 20:52:58.588882872 +0000 UTC m=+1418.592841597"
Nov 03 20:52:58.700019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:58.699930    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg" podStartSLOduration=1.6999092660000001 podStartE2EDuration="1.699909266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:58.693566483 +0000 UTC m=+1418.697525227" watchObservedRunningTime="2026-11-03 20:52:58.699909266 +0000 UTC m=+1418.703867991"
Nov 03 20:52:59.047571 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:59.047527    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"]
Nov 03 20:52:59.054477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:59.054446    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1eb5db4-jpslc"]
Nov 03 20:52:59.105270 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:59.105236    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:59.308691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:59.308589    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:59.498380 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:52:59.498345    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:00.112729 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.112698    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c99c1a-549b-4580-b9e3-8f32fbb4d56b" path="/var/lib/kubelet/pods/68c99c1a-549b-4580-b9e3-8f32fbb4d56b/volumes"
Nov 03 20:53:00.168484 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.168450    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"]
Nov 03 20:53:00.173973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.173947    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.228357 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.228306    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 20:53:00.228357 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.228346    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 20:53:00.228688 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.228668    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 20:53:00.228745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.228693    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4mvmz\""
Nov 03 20:53:00.281994 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.281951    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"]
Nov 03 20:53:00.330728 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.330695    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.330878 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.330843    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2248m\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-kube-api-access-2248m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.330942 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.330876    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/103c0f10-3b8e-4747-b25a-84634edc6a3a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.330942 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.330935    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.331139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.331104    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.331283 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.331175    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.331352 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.331300    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.432554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.432473    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.432716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.432568    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.432716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.432670    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2248m\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-kube-api-access-2248m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.432716 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.432700    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/103c0f10-3b8e-4747-b25a-84634edc6a3a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.432873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.432757    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.432873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.432808    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.432873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.432846    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.433844 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.433813    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.435402 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.435379    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.435515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.435466    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.435515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.435468    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/103c0f10-3b8e-4747-b25a-84634edc6a3a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.435912 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.435895    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.487284 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.487244    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.487462 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.487333    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2248m\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-kube-api-access-2248m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4nvbf\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.493195 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.493168    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:00.936202 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:00.936166    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"]
Nov 03 20:53:00.938258 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:53:00.938227    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103c0f10_3b8e_4747_b25a_84634edc6a3a.slice/crio-1c31be20b767acc8bfb5126365dab8b857fdb1714c145bfbdbca3a0a4387a880 WatchSource:0}: Error finding container 1c31be20b767acc8bfb5126365dab8b857fdb1714c145bfbdbca3a0a4387a880: Status 404 returned error can't find the container with id 1c31be20b767acc8bfb5126365dab8b857fdb1714c145bfbdbca3a0a4387a880
Nov 03 20:53:01.518157 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:01.518121    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf" event={"ID":"103c0f10-3b8e-4747-b25a-84634edc6a3a","Type":"ContainerStarted","Data":"1c31be20b767acc8bfb5126365dab8b857fdb1714c145bfbdbca3a0a4387a880"}
Nov 03 20:53:01.608242 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:01.608199    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:02.393931 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.393892    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"]
Nov 03 20:53:02.398949 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.398922    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.450442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.450409    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"]
Nov 03 20:53:02.490339 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.490302    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:53:02.490339 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.490333    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:53:02.567999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.567964    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.568210 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.568080    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.568276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.568257    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfjz\" (UniqueName: \"kubernetes.io/projected/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-kube-api-access-vgfjz\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.669556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.669475    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfjz\" (UniqueName: \"kubernetes.io/projected/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-kube-api-access-vgfjz\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.669785 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.669674    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.669785 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.669751    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.670493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.670462    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.672657 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.672627    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:02.726249 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:02.726217    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfjz\" (UniqueName: \"kubernetes.io/projected/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-kube-api-access-vgfjz\") pod \"maas-api-revoke-sub-3cccdecc-9bfhx\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:03.025811 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.025730    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:03.421809 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.421781    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"]
Nov 03 20:53:03.424576 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:53:03.424547    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a49a61_1d0e_4b66_90b6_3c4936eb2283.slice/crio-c3a8a0e943156ad0006d0332139e32fa9f5ba7b80cbc201d324b3e6034e748f2 WatchSource:0}: Error finding container c3a8a0e943156ad0006d0332139e32fa9f5ba7b80cbc201d324b3e6034e748f2: Status 404 returned error can't find the container with id c3a8a0e943156ad0006d0332139e32fa9f5ba7b80cbc201d324b3e6034e748f2
Nov 03 20:53:03.543730 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.543682    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx" event={"ID":"f0a49a61-1d0e-4b66-90b6-3c4936eb2283","Type":"ContainerStarted","Data":"c3a8a0e943156ad0006d0332139e32fa9f5ba7b80cbc201d324b3e6034e748f2"}
Nov 03 20:53:03.627903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.627861    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:03.729136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.729036    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"]
Nov 03 20:53:03.744692 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.738128    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:03.787387 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.787335    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"]
Nov 03 20:53:03.909171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.901428    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788b1908-6838-497f-9b10-66dd33942d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:03.909171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.901528    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nml9\" (UniqueName: \"kubernetes.io/projected/788b1908-6838-497f-9b10-66dd33942d47-kube-api-access-4nml9\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:03.909171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:03.901618    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788b1908-6838-497f-9b10-66dd33942d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:04.007768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.007426    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788b1908-6838-497f-9b10-66dd33942d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:04.007768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.007510    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nml9\" (UniqueName: \"kubernetes.io/projected/788b1908-6838-497f-9b10-66dd33942d47-kube-api-access-4nml9\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:04.007768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.007589    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788b1908-6838-497f-9b10-66dd33942d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:04.008442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.008413    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788b1908-6838-497f-9b10-66dd33942d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:04.011273 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.011250    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788b1908-6838-497f-9b10-66dd33942d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:04.071398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.071345    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nml9\" (UniqueName: \"kubernetes.io/projected/788b1908-6838-497f-9b10-66dd33942d47-kube-api-access-4nml9\") pod \"maas-api-revoke-sub-44c9b013-fqzzf\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:04.369442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.369406    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:04.573235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.572996    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:04.607533 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.605763    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx" event={"ID":"f0a49a61-1d0e-4b66-90b6-3c4936eb2283","Type":"ContainerStarted","Data":"b503c2493b6263f12a9dc0ac835d9d96e1a19a2c06d07911c30fcf44b6d32cb1"}
Nov 03 20:53:04.777211 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.777151    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx" podStartSLOduration=2.7771282250000002 podStartE2EDuration="2.777128225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:04.762470871 +0000 UTC m=+1424.766429599" watchObservedRunningTime="2026-11-03 20:53:04.777128225 +0000 UTC m=+1424.781086953"
Nov 03 20:53:04.851496 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.851453    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:04.889202 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:04.889168    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"]
Nov 03 20:53:05.126673 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.126612    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:05.126775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.126680    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:05.363665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.363628    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:05.511234 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.511148    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:05.616906 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.616866    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf" event={"ID":"103c0f10-3b8e-4747-b25a-84634edc6a3a","Type":"ContainerStarted","Data":"672c73336600261964330cec63d1713ce59f54dba584c0e0067eea26cc20e814"}
Nov 03 20:53:05.617460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.616937    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:05.618978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.618942    5196 generic.go:356] "Generic (PLEG): container finished" podID="f0a49a61-1d0e-4b66-90b6-3c4936eb2283" containerID="b503c2493b6263f12a9dc0ac835d9d96e1a19a2c06d07911c30fcf44b6d32cb1" exitCode=0
Nov 03 20:53:05.619137 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.619028    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx" event={"ID":"f0a49a61-1d0e-4b66-90b6-3c4936eb2283","Type":"ContainerDied","Data":"b503c2493b6263f12a9dc0ac835d9d96e1a19a2c06d07911c30fcf44b6d32cb1"}
Nov 03 20:53:05.620945 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.620922    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf" event={"ID":"788b1908-6838-497f-9b10-66dd33942d47","Type":"ContainerStarted","Data":"23a4957e57abd492f2f6538a5631445907fc4b3a0101610281e8603158f4e5a0"}
Nov 03 20:53:05.621059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.620960    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf" event={"ID":"788b1908-6838-497f-9b10-66dd33942d47","Type":"ContainerStarted","Data":"50bd79e16cecaff2e0cbc8777eae95e03df1529686edc3fff75471c90d05b886"}
Nov 03 20:53:05.729685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.729623    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf" podStartSLOduration=1.547444911 podStartE2EDuration="5.729603334s" totalImagesPullingTime="4.182158423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:05.727344342 +0000 UTC m=+1425.731303087" watchObservedRunningTime="2026-11-03 20:53:05.729603334 +0000 UTC m=+1425.733562059"
Nov 03 20:53:05.866343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:05.866278    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf" podStartSLOduration=2.866259181 podStartE2EDuration="2.866259181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:05.858877892 +0000 UTC m=+1425.862836640" watchObservedRunningTime="2026-11-03 20:53:05.866259181 +0000 UTC m=+1425.870217905"
Nov 03 20:53:06.635142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.635103    5196 generic.go:356] "Generic (PLEG): container finished" podID="788b1908-6838-497f-9b10-66dd33942d47" containerID="23a4957e57abd492f2f6538a5631445907fc4b3a0101610281e8603158f4e5a0" exitCode=0
Nov 03 20:53:06.635667 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.635168    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf" event={"ID":"788b1908-6838-497f-9b10-66dd33942d47","Type":"ContainerDied","Data":"23a4957e57abd492f2f6538a5631445907fc4b3a0101610281e8603158f4e5a0"}
Nov 03 20:53:06.638703 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.638672    5196 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-4nvbf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:53:06.638825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.638733    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf" podUID="103c0f10-3b8e-4747-b25a-84634edc6a3a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:53:06.868231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.868196    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:06.960364 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.960267    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:06.974818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.974781    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-openshift-ca\") pod \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") "
Nov 03 20:53:06.975133 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.975002    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-kube-api-access-vgfjz\" (UniqueName: \"kubernetes.io/projected/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-kube-api-access-vgfjz\") pod \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") "
Nov 03 20:53:06.975246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.975087    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-xks-ca\") pod \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\" (UID: \"f0a49a61-1d0e-4b66-90b6-3c4936eb2283\") "
Nov 03 20:53:06.976315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.976291    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-openshift-ca" pod "f0a49a61-1d0e-4b66-90b6-3c4936eb2283" (UID: "f0a49a61-1d0e-4b66-90b6-3c4936eb2283"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:06.978447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.978408    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-xks-ca" pod "f0a49a61-1d0e-4b66-90b6-3c4936eb2283" (UID: "f0a49a61-1d0e-4b66-90b6-3c4936eb2283"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:53:06.978769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:06.978671    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-kube-api-access-vgfjz" pod "f0a49a61-1d0e-4b66-90b6-3c4936eb2283" (UID: "f0a49a61-1d0e-4b66-90b6-3c4936eb2283"). InnerVolumeSpecName "kube-api-access-vgfjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:07.078006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.077965    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgfjz\" (UniqueName: \"kubernetes.io/projected/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-kube-api-access-vgfjz\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:07.078006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.078004    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:07.078006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.078016    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a49a61-1d0e-4b66-90b6-3c4936eb2283-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:07.647503 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.647470    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"
Nov 03 20:53:07.647920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.647503    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx" event={"ID":"f0a49a61-1d0e-4b66-90b6-3c4936eb2283","Type":"ContainerDied","Data":"c3a8a0e943156ad0006d0332139e32fa9f5ba7b80cbc201d324b3e6034e748f2"}
Nov 03 20:53:07.647920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.647566    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3a8a0e943156ad0006d0332139e32fa9f5ba7b80cbc201d324b3e6034e748f2"
Nov 03 20:53:07.825001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.824973    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:07.889848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.889817    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788b1908-6838-497f-9b10-66dd33942d47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788b1908-6838-497f-9b10-66dd33942d47-maas-api-openshift-ca\") pod \"788b1908-6838-497f-9b10-66dd33942d47\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") "
Nov 03 20:53:07.890020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.889932    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788b1908-6838-497f-9b10-66dd33942d47-kube-api-access-4nml9\" (UniqueName: \"kubernetes.io/projected/788b1908-6838-497f-9b10-66dd33942d47-kube-api-access-4nml9\") pod \"788b1908-6838-497f-9b10-66dd33942d47\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") "
Nov 03 20:53:07.890020 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.889970    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788b1908-6838-497f-9b10-66dd33942d47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788b1908-6838-497f-9b10-66dd33942d47-maas-api-xks-ca\") pod \"788b1908-6838-497f-9b10-66dd33942d47\" (UID: \"788b1908-6838-497f-9b10-66dd33942d47\") "
Nov 03 20:53:07.890401 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.890343    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788b1908-6838-497f-9b10-66dd33942d47-maas-api-openshift-ca" pod "788b1908-6838-497f-9b10-66dd33942d47" (UID: "788b1908-6838-497f-9b10-66dd33942d47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:07.891171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.891043    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788b1908-6838-497f-9b10-66dd33942d47-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:07.892672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.892651    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788b1908-6838-497f-9b10-66dd33942d47-kube-api-access-4nml9" pod "788b1908-6838-497f-9b10-66dd33942d47" (UID: "788b1908-6838-497f-9b10-66dd33942d47"). InnerVolumeSpecName "kube-api-access-4nml9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:07.892745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.892698    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788b1908-6838-497f-9b10-66dd33942d47-maas-api-xks-ca" pod "788b1908-6838-497f-9b10-66dd33942d47" (UID: "788b1908-6838-497f-9b10-66dd33942d47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:53:07.993082 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.993045    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nml9\" (UniqueName: \"kubernetes.io/projected/788b1908-6838-497f-9b10-66dd33942d47-kube-api-access-4nml9\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:07.993082 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:07.993076    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788b1908-6838-497f-9b10-66dd33942d47-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:08.660926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:08.660899    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"
Nov 03 20:53:08.660926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:08.660906    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf" event={"ID":"788b1908-6838-497f-9b10-66dd33942d47","Type":"ContainerDied","Data":"50bd79e16cecaff2e0cbc8777eae95e03df1529686edc3fff75471c90d05b886"}
Nov 03 20:53:08.661530 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:08.660953    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50bd79e16cecaff2e0cbc8777eae95e03df1529686edc3fff75471c90d05b886"
Nov 03 20:53:09.497344 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:09.497299    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:53:09.505616 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:09.498374    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:53:09.636929 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:09.636898    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 20:53:11.775437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:11.775402    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:12.954916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:12.954878    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:13.145849 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:13.145808    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:15.933903 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:15.933869    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:16.124117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:16.123064    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:16.436352 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:16.436313    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:16.677115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:16.674601    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:16.827477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:16.827439    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:17.426372 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:17.426339    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:17.597171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:17.597086    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:17.754799 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:17.754715    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:18.057196 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:18.057078    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:18.489919 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:53:18.489881    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:53:18.549011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:18.548971    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:20.551325 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:20.551277    5196 scope.go:122] "RemoveContainer" containerID="dbed79e3c2b96ef7f1d295ab371a097ef2930f21eaacf9c5d09652f2e31ae169"
Nov 03 20:53:20.661037 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:20.661012    5196 scope.go:122] "RemoveContainer" containerID="c0359997e2db51fb7ea51b91b5f5b023dc9af32faf4926680c6d6f935facbc23"
Nov 03 20:53:20.791123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:20.789965    5196 scope.go:122] "RemoveContainer" containerID="179de4736fb202aa07a4f7bfcf73906738a7158f5850207ff1fc966c01019457"
Nov 03 20:53:21.216059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.216027    5196 scope.go:122] "RemoveContainer" containerID="6378721dd83aaeebbbcfc4c11c625f1b6d8285392f2568b3f03c61a0f1b8e420"
Nov 03 20:53:21.313235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.313192    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:21.371742 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.371252    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:21.372318 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.372014    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:21.432224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.418404    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": context deadline exceeded"
Nov 03 20:53:21.572167 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.568488    5196 scope.go:122] "RemoveContainer" containerID="4595a15238223684d85ab8c2a86d5bc342d3459c74c5ea36c2e8c3e962358d02"
Nov 03 20:53:21.680181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.678192    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:21.680181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.678656    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 03 20:53:21.741215 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.741154    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 20:53:21.789174 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.779768    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:21.818469 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:21.818439    5196 scope.go:122] "RemoveContainer" containerID="0283ff85e3f6f3583e702cfdab48d757989dea255c73e45eae56fd5a90867c23"
Nov 03 20:53:22.068386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.062281    5196 scope.go:122] "RemoveContainer" containerID="d36c7d128d74dbdf6bc2bcfc304d631c989c462a1ec51b61e775daff77f88740"
Nov 03 20:53:22.291371 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.291318    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/health\": context deadline exceeded"
Nov 03 20:53:22.345236 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.322176    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:22.345236 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.327834    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:22.396422 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.384027    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:22.397069 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.397024    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 03 20:53:22.419297 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.408990    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:53:22.419297 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.409080    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:53:22.419297 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.409159    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:53:22.450461 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.448613    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": context deadline exceeded"
Nov 03 20:53:22.484755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.484710    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:22.916121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:22.914729    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:53:23.173179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.156478    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:23.303816 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.300237    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:23.303816 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.303199    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:23.332940 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.332892    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:23.408587 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.385694    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 20:53:23.453370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.453258    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:23.454070 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.454038    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:23.458218 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.458171    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:23.483136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.476016    5196 scope.go:122] "RemoveContainer" containerID="3b6db91887ec5b687d6d9514b3122d9211414aa7ed94971c3a51fc9081dc8bed"
Nov 03 20:53:23.504172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.488206    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": context deadline exceeded"
Nov 03 20:53:23.504172 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:53:23.500906    5196 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.092s"
Nov 03 20:53:23.504172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.500995    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:53:23.504172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.501134    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:53:23.504172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.501172    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:53:23.504172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.501204    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:53:23.504172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.501241    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:53:23.932371 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.932321    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"]
Nov 03 20:53:23.934127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.934075    5196 cpu_manager.go:414] "containerMap: removing container" podUID="788b1908-6838-497f-9b10-66dd33942d47" containerName="revoke-keys"
Nov 03 20:53:23.957584 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.957364    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788b1908-6838-497f-9b10-66dd33942d47" containerName="revoke-keys"
Nov 03 20:53:23.957965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.957943    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f0a49a61-1d0e-4b66-90b6-3c4936eb2283" containerName="revoke-keys"
Nov 03 20:53:23.958156 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.958140    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a49a61-1d0e-4b66-90b6-3c4936eb2283" containerName="revoke-keys"
Nov 03 20:53:23.958669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.958637    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="788b1908-6838-497f-9b10-66dd33942d47" containerName="revoke-keys"
Nov 03 20:53:24.004477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.004433    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a49a61-1d0e-4b66-90b6-3c4936eb2283" containerName="revoke-keys"
Nov 03 20:53:24.016500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.957303    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:53:24.018257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.994920    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": context deadline exceeded"
Nov 03 20:53:24.019647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.019457    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:24.047269 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:23.994306    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded"
Nov 03 20:53:24.047540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.044032    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"]
Nov 03 20:53:24.047856 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.047787    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:53:24.048048 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.048032    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:53:24.051705 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.044186    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.078886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.077869    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 20:53:24.126825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.126202    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.126825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.126270    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ffa19a-6140-46db-b407-4be9e07da949-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.126825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.126359    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rgcr\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-kube-api-access-4rgcr\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.126825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.126443    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.126825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.126690    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ffa19a-6140-46db-b407-4be9e07da949-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.126825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.126779    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.127324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.126879    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.127324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.126916    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.127324 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.127034    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.182478 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.182348    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:53:24.182748 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.182703    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:24.201494 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.201466    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 20:53:24.201723 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.201710    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 20:53:24.206863 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.206839    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz"
Nov 03 20:53:24.220410 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.219468    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:24.220606 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.220420    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.228080    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230114    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ffa19a-6140-46db-b407-4be9e07da949-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230217    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230299    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230349    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230431    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230572    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230603    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ffa19a-6140-46db-b407-4be9e07da949-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230678    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rgcr\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-kube-api-access-4rgcr\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.230779    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.231452    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.233826    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.234144    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.239002    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.242117 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.239616    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ffa19a-6140-46db-b407-4be9e07da949-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.267556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.264543    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 20:53:24.271819 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.271787    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 20:53:24.272178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.272162    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 20:53:24.293978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.291756    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ffa19a-6140-46db-b407-4be9e07da949-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.300569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.298536    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:24.300569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.299537    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 20:53:24.309800 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.309763    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.310263 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.310056    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rgcr\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-kube-api-access-4rgcr\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.319954 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.318576    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:24.546715 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:24.546671    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:25.017796 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:25.017721    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:25.018259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:25.017924    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:25.018259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:25.018034    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:25.023086 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:25.022712    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"]
Nov 03 20:53:25.311552 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:25.310317    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" event={"ID":"d1ffa19a-6140-46db-b407-4be9e07da949","Type":"ContainerStarted","Data":"3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd"}
Nov 03 20:53:25.311552 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:25.310392    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" event={"ID":"d1ffa19a-6140-46db-b407-4be9e07da949","Type":"ContainerStarted","Data":"b4271c2332774237ead504ecbf7fdc1b9d8c0166c8f61d7aea16437ae7584f0a"}
Nov 03 20:53:25.546867 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:25.546829    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:26.562129 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:26.554650    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:53:26.562129 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:26.554725    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:53:26.622331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:26.622286    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:26.813247 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:26.813147    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:27.556121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:27.547267    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:53:27.556121 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:27.547342    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:53:28.518417 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:28.518379    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:28.556142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:28.554536    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:53:28.556142 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:28.554604    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:53:28.831533 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:28.831486    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:28.985326 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:28.985271    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:29.431397 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:29.431356    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:29.566978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:29.566937    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:53:29.567594 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:29.567564    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:53:29.598813 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:29.598772    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:29.928130 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:29.926080    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podStartSLOduration=6.926053749 podStartE2EDuration="6.926053749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:25.887282713 +0000 UTC m=+1445.891241439" watchObservedRunningTime="2026-11-03 20:53:29.926053749 +0000 UTC m=+1449.930012475"
Nov 03 20:53:29.957116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:29.957050    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"]
Nov 03 20:53:29.994223 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:29.974231    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:30.032001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.031696    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqfmw\" (UniqueName: \"kubernetes.io/projected/421e65d1-3d4a-4578-bcd2-3d942f7f0082-kube-api-access-qqfmw\") pod \"payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2\" (UID: \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\") " pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:30.032001 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.031791    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421e65d1-3d4a-4578-bcd2-3d942f7f0082-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2\" (UID: \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\") " pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:30.119350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.110400    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e3e5cd62\""
Nov 03 20:53:30.119350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.110540    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e3e5cd62-dockercfg-bx96j\""
Nov 03 20:53:30.182182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.136265    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfmw\" (UniqueName: \"kubernetes.io/projected/421e65d1-3d4a-4578-bcd2-3d942f7f0082-kube-api-access-qqfmw\") pod \"payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2\" (UID: \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\") " pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:30.182182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.136377    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421e65d1-3d4a-4578-bcd2-3d942f7f0082-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2\" (UID: \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\") " pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:30.182182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.137808    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421e65d1-3d4a-4578-bcd2-3d942f7f0082-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2\" (UID: \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\") " pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:30.313164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.310797    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfmw\" (UniqueName: \"kubernetes.io/projected/421e65d1-3d4a-4578-bcd2-3d942f7f0082-kube-api-access-qqfmw\") pod \"payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2\" (UID: \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\") " pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:30.317829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.316139    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"]
Nov 03 20:53:30.467522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.467423    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:30.562120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.561192    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:53:30.562120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.561258    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:53:30.909429 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:30.909399    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"]
Nov 03 20:53:31.143120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.136178    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"]
Nov 03 20:53:31.164884 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.164787    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.190408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.189436    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg44q\" (UniqueName: \"kubernetes.io/projected/3b411414-ab81-41dc-ac21-0823b35e6200-kube-api-access-xg44q\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.190737 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.190713    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.191155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.191134    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.196453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.196410    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"]
Nov 03 20:53:31.215168 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.214674    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:53:31.215168 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.214838    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:53:31.293436 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.293386    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.293662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.293513    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg44q\" (UniqueName: \"kubernetes.io/projected/3b411414-ab81-41dc-ac21-0823b35e6200-kube-api-access-xg44q\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.293662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.293569    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.294415 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.294381    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.296348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.296325    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.347689 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.347652    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg44q\" (UniqueName: \"kubernetes.io/projected/3b411414-ab81-41dc-ac21-0823b35e6200-kube-api-access-xg44q\") pod \"maas-api-revoke-sub-f7f1ad4e-hhfxv\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.491446 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.491354    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:31.498532 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.498496    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2" event={"ID":"421e65d1-3d4a-4578-bcd2-3d942f7f0082","Type":"ContainerStarted","Data":"18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329"}
Nov 03 20:53:31.498920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.498902    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:31.499031 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.498939    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2" event={"ID":"421e65d1-3d4a-4578-bcd2-3d942f7f0082","Type":"ContainerStarted","Data":"2c9281d24eec87d31b5a19370e23e66d2b238272f40378eaab3656bc24bce09e"}
Nov 03 20:53:31.547336 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.547295    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:53:31.547535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.547369    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:53:31.625967 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.625897    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2" podStartSLOduration=2.6258756549999998 podStartE2EDuration="2.625875655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:31.624870768 +0000 UTC m=+1451.628829493" watchObservedRunningTime="2026-11-03 20:53:31.625875655 +0000 UTC m=+1451.629834379"
Nov 03 20:53:31.900109 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:31.899613    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"]
Nov 03 20:53:31.908301 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:53:31.908265    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b411414_ab81_41dc_ac21_0823b35e6200.slice/crio-7c4db98ad8e6634f4955ce8a27781b6e05223f4782401de374fe780514abab26 WatchSource:0}: Error finding container 7c4db98ad8e6634f4955ce8a27781b6e05223f4782401de374fe780514abab26: Status 404 returned error can't find the container with id 7c4db98ad8e6634f4955ce8a27781b6e05223f4782401de374fe780514abab26
Nov 03 20:53:32.515926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:32.515822    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv" event={"ID":"3b411414-ab81-41dc-ac21-0823b35e6200","Type":"ContainerStarted","Data":"d55a9945d73ba02909af75cdb288e0420244b45397ab98a861c407063654825a"}
Nov 03 20:53:32.515926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:32.515898    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv" event={"ID":"3b411414-ab81-41dc-ac21-0823b35e6200","Type":"ContainerStarted","Data":"7c4db98ad8e6634f4955ce8a27781b6e05223f4782401de374fe780514abab26"}
Nov 03 20:53:32.631083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:32.631016    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv" podStartSLOduration=1.6309997 podStartE2EDuration="1.6309997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:32.624531602 +0000 UTC m=+1452.628490379" watchObservedRunningTime="2026-11-03 20:53:32.6309997 +0000 UTC m=+1452.634958423"
Nov 03 20:53:32.766583 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:32.766501    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:32.768310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:32.768272    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:32.769240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:32.769210    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 20:53:33.420310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:33.420220    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:33.578133 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:33.576516    5196 generic.go:356] "Generic (PLEG): container finished" podID="3b411414-ab81-41dc-ac21-0823b35e6200" containerID="d55a9945d73ba02909af75cdb288e0420244b45397ab98a861c407063654825a" exitCode=0
Nov 03 20:53:33.578739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:33.578711    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv" event={"ID":"3b411414-ab81-41dc-ac21-0823b35e6200","Type":"ContainerDied","Data":"d55a9945d73ba02909af75cdb288e0420244b45397ab98a861c407063654825a"}
Nov 03 20:53:33.581585 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:33.581544    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:33.738421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:33.738279    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:34.763506 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.763478    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:34.816127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.816058    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-openshift-ca\") pod \"3b411414-ab81-41dc-ac21-0823b35e6200\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") "
Nov 03 20:53:34.816300 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.816182    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-xks-ca\") pod \"3b411414-ab81-41dc-ac21-0823b35e6200\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") "
Nov 03 20:53:34.816300 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.816236    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b411414-ab81-41dc-ac21-0823b35e6200-kube-api-access-xg44q\" (UniqueName: \"kubernetes.io/projected/3b411414-ab81-41dc-ac21-0823b35e6200-kube-api-access-xg44q\") pod \"3b411414-ab81-41dc-ac21-0823b35e6200\" (UID: \"3b411414-ab81-41dc-ac21-0823b35e6200\") "
Nov 03 20:53:34.816426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.816403    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-openshift-ca" pod "3b411414-ab81-41dc-ac21-0823b35e6200" (UID: "3b411414-ab81-41dc-ac21-0823b35e6200"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:34.816932 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.816909    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:34.818543 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.818515    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-xks-ca" pod "3b411414-ab81-41dc-ac21-0823b35e6200" (UID: "3b411414-ab81-41dc-ac21-0823b35e6200"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:53:34.818645 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.818553    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b411414-ab81-41dc-ac21-0823b35e6200-kube-api-access-xg44q" pod "3b411414-ab81-41dc-ac21-0823b35e6200" (UID: "3b411414-ab81-41dc-ac21-0823b35e6200"). InnerVolumeSpecName "kube-api-access-xg44q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:34.918579 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.918492    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b411414-ab81-41dc-ac21-0823b35e6200-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:34.918579 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:34.918537    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg44q\" (UniqueName: \"kubernetes.io/projected/3b411414-ab81-41dc-ac21-0823b35e6200-kube-api-access-xg44q\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:35.018496 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:35.018424    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:35.216909 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:35.216796    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:35.600595 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:35.600565    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"
Nov 03 20:53:35.600790 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:35.600600    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv" event={"ID":"3b411414-ab81-41dc-ac21-0823b35e6200","Type":"ContainerDied","Data":"7c4db98ad8e6634f4955ce8a27781b6e05223f4782401de374fe780514abab26"}
Nov 03 20:53:35.600790 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:35.600662    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c4db98ad8e6634f4955ce8a27781b6e05223f4782401de374fe780514abab26"
Nov 03 20:53:38.153308 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.153270    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:38.808255 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.808207    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"]
Nov 03 20:53:38.810162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.810126    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3b411414-ab81-41dc-ac21-0823b35e6200" containerName="revoke-keys"
Nov 03 20:53:38.810162 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.810161    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b411414-ab81-41dc-ac21-0823b35e6200" containerName="revoke-keys"
Nov 03 20:53:38.810876 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.810733    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b411414-ab81-41dc-ac21-0823b35e6200" containerName="revoke-keys"
Nov 03 20:53:38.831531 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.831488    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.901537 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.901499    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"]
Nov 03 20:53:38.981961 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.981916    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.982203 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.982033    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.982203 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.982135    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.982331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.982201    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.982331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.982278    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scmn\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-kube-api-access-7scmn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.982432 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.982343    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.982432 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.982359    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee909c-37c4-4418-aabf-6ffc3cfe4267-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.982432 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.982374    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:38.982583 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:38.982476    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084307    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084345    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee909c-37c4-4418-aabf-6ffc3cfe4267-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084368    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084439    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084621    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084693    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084844 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084758    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084844 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084807    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.084952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084866    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scmn\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-kube-api-access-7scmn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.085015 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.084975    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-certs\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.085077 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.085046    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.085227 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.085203    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-credential-socket\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.085227 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.085219    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee909c-37c4-4418-aabf-6ffc3cfe4267-istiod-ca-cert\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.085333 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.085209    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-data\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.087250 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.087222    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-envoy\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.087781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.087758    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-podinfo\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.139685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.139653    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-token\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.139846 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.139714    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scmn\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-kube-api-access-7scmn\") pod \"e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") " pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.154262 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.154232    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:39.598457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.598418    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"]
Nov 03 20:53:39.610998 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:53:39.610428    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedee909c_37c4_4418_aabf_6ffc3cfe4267.slice/crio-cc6daa93925f6e9f71badee0a69ef8c844b41c217c98a6332000c0f775fc963c WatchSource:0}: Error finding container cc6daa93925f6e9f71badee0a69ef8c844b41c217c98a6332000c0f775fc963c: Status 404 returned error can't find the container with id cc6daa93925f6e9f71badee0a69ef8c844b41c217c98a6332000c0f775fc963c
Nov 03 20:53:39.613662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.613626    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:39.613781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.613696    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:39.613781 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.613730    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:39.671217 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:39.671078    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" event={"ID":"edee909c-37c4-4418-aabf-6ffc3cfe4267","Type":"ContainerStarted","Data":"cc6daa93925f6e9f71badee0a69ef8c844b41c217c98a6332000c0f775fc963c"}
Nov 03 20:53:40.570905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.570871    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"]
Nov 03 20:53:40.577208 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.577180    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.600560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.600524    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"]
Nov 03 20:53:40.606363 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.606338    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.685059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.684082    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"]
Nov 03 20:53:40.695127 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.694668    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" event={"ID":"edee909c-37c4-4418-aabf-6ffc3cfe4267","Type":"ContainerStarted","Data":"9c756a866f02c74d2a08f2b3ce0f6192d4793230dbfcbbf2992efecf561bb2e2"}
Nov 03 20:53:40.698137 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.696605    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"]
Nov 03 20:53:40.715110 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715062    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.715275 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715180    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.715275 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715207    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tlq5\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-kube-api-access-7tlq5\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.715275 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715241    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7aefef8-4211-4e77-9949-2223b25363ca-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.715275 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715262    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.715626 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715312    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.715626 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715460    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.715626 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715607    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.715803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715663    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.715803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715703    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.715803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715750    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.715803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715791    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.716018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715933    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.716018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.715975    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.716018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.716000    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.716200 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.716037    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2zg\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-kube-api-access-bz2zg\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.716200 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.716078    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.716409 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.716385    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7aefef8-4211-4e77-9949-2223b25363ca-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.818707 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818669    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.818886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818713    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.818886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818737    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.818886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818762    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.818886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818787    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.818886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818817    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.818886 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818841    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818928    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818954    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818970    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.818991    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2zg\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-kube-api-access-bz2zg\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819024    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819079    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7aefef8-4211-4e77-9949-2223b25363ca-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819154    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819164    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-data\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819222    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819239    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tlq5\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-kube-api-access-7tlq5\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.819265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819257    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7aefef8-4211-4e77-9949-2223b25363ca-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.820535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819277    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.820535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819898    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-certs\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.820535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.819929    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.820535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.820322    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.820535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.820455    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.820820 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.820547    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.820820 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.820671    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-credential-socket\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.820820 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.820677    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.821011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.820880    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7aefef8-4211-4e77-9949-2223b25363ca-istiod-ca-cert\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.821011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.820957    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.822956 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.822931    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.823408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.823385    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.823558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.823537    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7aefef8-4211-4e77-9949-2223b25363ca-istio-podinfo\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.823837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.823815    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-envoy\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.874848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.874160    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-istio-token\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.874848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.874327    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2zg\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-kube-api-access-bz2zg\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.874848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.874655    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.875617 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.875594    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tlq5\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-kube-api-access-7tlq5\") pod \"e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") " pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:40.893762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.893739    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:40.920895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.920843    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podStartSLOduration=2.920828285 podStartE2EDuration="2.920828285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:40.918234229 +0000 UTC m=+1460.922192974" watchObservedRunningTime="2026-11-03 20:53:40.920828285 +0000 UTC m=+1460.924787009"
Nov 03 20:53:40.924145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:40.924084    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:41.167140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.159924    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:41.167140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.160054    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 20:53:41.167140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.160124    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 20:53:41.365604 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:53:41.365500    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f17ae4d_acf3_4475_b7d2_1c9e8704a0de.slice/crio-af2de49713e91fe0e2fc9555e6307c4b545a8da01d6ec1b402a7e758e7a9d77e WatchSource:0}: Error finding container af2de49713e91fe0e2fc9555e6307c4b545a8da01d6ec1b402a7e758e7a9d77e: Status 404 returned error can't find the container with id af2de49713e91fe0e2fc9555e6307c4b545a8da01d6ec1b402a7e758e7a9d77e
Nov 03 20:53:41.368012 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.367989    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"]
Nov 03 20:53:41.368572 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.368534    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:41.368669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.368616    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:41.368669 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.368660    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:41.377651 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.377625    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"]
Nov 03 20:53:41.406478 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:53:41.406451    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7aefef8_4211_4e77_9949_2223b25363ca.slice/crio-5947b768a2c381b323de3a8d609c5dc522f465c660e96dfc626aa86a748670d3 WatchSource:0}: Error finding container 5947b768a2c381b323de3a8d609c5dc522f465c660e96dfc626aa86a748670d3: Status 404 returned error can't find the container with id 5947b768a2c381b323de3a8d609c5dc522f465c660e96dfc626aa86a748670d3
Nov 03 20:53:41.409310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.409278    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:41.409389 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.409353    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:41.409389 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.409382    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:53:41.714199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.710246    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" event={"ID":"b7aefef8-4211-4e77-9949-2223b25363ca","Type":"ContainerStarted","Data":"47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08"}
Nov 03 20:53:41.714199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.710315    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" event={"ID":"b7aefef8-4211-4e77-9949-2223b25363ca","Type":"ContainerStarted","Data":"5947b768a2c381b323de3a8d609c5dc522f465c660e96dfc626aa86a748670d3"}
Nov 03 20:53:41.714199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.712265    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" event={"ID":"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de","Type":"ContainerStarted","Data":"47808c30d8a8e4aecc3db4ba394dbf7e335d3733e56fd5418bd6c122a563aee0"}
Nov 03 20:53:41.714199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.712307    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" event={"ID":"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de","Type":"ContainerStarted","Data":"af2de49713e91fe0e2fc9555e6307c4b545a8da01d6ec1b402a7e758e7a9d77e"}
Nov 03 20:53:41.774635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.774490    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:41.825647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.825591    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podStartSLOduration=1.8255715719999999 podStartE2EDuration="1.825571572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:41.824387738 +0000 UTC m=+1461.828346463" watchObservedRunningTime="2026-11-03 20:53:41.825571572 +0000 UTC m=+1461.829530287"
Nov 03 20:53:41.897306 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.894511    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:41.924802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.924770    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:41.931762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.931727    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:41.959975 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:41.959911    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podStartSLOduration=1.959889509 podStartE2EDuration="1.959889509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:41.941759518 +0000 UTC m=+1461.945718259" watchObservedRunningTime="2026-11-03 20:53:41.959889509 +0000 UTC m=+1461.963848235"
Nov 03 20:53:42.156769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:42.156621    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 20:53:42.160216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:42.156914    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 20:53:42.424645 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:42.424538    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:42.915482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:42.915435    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:53:42.917517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:42.916030    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 20:53:42.931155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:42.930330    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:53:42.931155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:42.930395    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:53:43.117421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:43.117385    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:43.161118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:43.160539    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 20:53:43.161118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:43.160610    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 20:53:43.924457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:43.924403    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:53:43.925893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:43.925862    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:53:43.926388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:43.926359    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:53:43.926768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:43.926737    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:53:43.926868 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:43.926791    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 20:53:44.083986 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:44.082841    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:44.284182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:44.283470    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 20:53:44.284182 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:44.283536    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 20:53:44.910118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:44.909243    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:53:44.910118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:44.909317    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 20:53:44.924840 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:44.924798    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:53:44.925413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:44.924856    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:53:45.163358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:45.163261    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 20:53:45.163634 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:45.163604    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 20:53:45.408307 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:45.408257    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:45.897670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:45.897625    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:53:45.897893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:45.897693    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 20:53:45.926531 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:45.926491    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:53:45.927172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:45.927140    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:53:46.157172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:46.156492    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 20:53:46.157172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:46.156553    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 20:53:46.897114 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:46.896628    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:53:46.897114 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:46.896710    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 20:53:46.925600 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:46.925558    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:53:46.925907 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:46.925870    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:53:47.684853 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:47.684813    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:47.704198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:47.687077    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:47.708422 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:47.708351    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 20:53:47.894408 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:47.894367    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:53:47.894608 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:47.894440    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 03 20:53:47.924763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:47.924727    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:53:47.924955 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:47.924777    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:53:48.343135 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:48.343078    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:48.496877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:48.496843    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:48.498057 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:53:48.498027    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:53:48.636902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:48.636802    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:48.925508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:48.925411    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:53:48.925508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:48.925473    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:53:49.334801 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:49.334747    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:49.337447 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:49.337420    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:49.340002 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:49.339965    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 20:53:49.929053 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:49.929022    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:49.929604 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:49.929445    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:49.930186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:49.930166    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 20:53:53.264495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:53.264463    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:58.078479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:53:58.078445    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:02.862192 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:02.862144    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:06.029741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:06.029696    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:54:07.764740 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:07.764691    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:07.968538 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:07.968496    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:08.875120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:08.872741    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:10.045382 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:10.045317    5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Nov 03 20:54:10.159120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:10.141232    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:10.278541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:10.244788    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:10.278541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:10.244910    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz" podUID="ed185af4-9540-4841-a9c3-bd14cf61e7e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:54:10.278541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:10.245013    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:10.278541 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:10.246292    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:10.279563 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:10.279517    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:10.539124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:10.508378    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:13.142370 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:13.142322    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:17.803513 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:17.803473    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:17.959059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:17.959018    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:18.185965 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:18.185930    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:18.488050 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:54:18.487966    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:54:20.366595 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:20.366568    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:54:20.373455 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:20.373432    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:54:20.376139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:20.376080    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:54:20.382406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:20.382386    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:54:22.755736 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:22.755694    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:24.469591 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:24.469558    5196 scope.go:122] "RemoveContainer" containerID="bd55a82148a8e89d94548b6ee1cc95b3f5ca366333e1d5aa1b305351133e919c"
Nov 03 20:54:24.483192 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:24.483084    5196 scope.go:122] "RemoveContainer" containerID="fdb1fc65f4a2db68c4eedc188013f40a753ab7a799c989623f6eb4581228e134"
Nov 03 20:54:24.496210 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:24.496188    5196 scope.go:122] "RemoveContainer" containerID="9cd51a450a6afbe1262ce566241ecd88e1eafe4ab78188834764cb7349896824"
Nov 03 20:54:24.510713 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:24.510685    5196 scope.go:122] "RemoveContainer" containerID="b401ee05f3d1899f4ac0052d61d2fab077cb37ba4baecf43872f087d828b6606"
Nov 03 20:54:24.523650 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:24.523625    5196 scope.go:122] "RemoveContainer" containerID="b9eb3e483af41c1672fefabb7e96e37bcc369b895eb2ddd16aea5f9d47d35e0b"
Nov 03 20:54:24.536574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:24.536551    5196 scope.go:122] "RemoveContainer" containerID="8525a42d42a0ec4b7ba8723f46aad8ddac4860be3d979a28274717f9b18c1980"
Nov 03 20:54:24.548025 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:24.548005    5196 scope.go:122] "RemoveContainer" containerID="641f7e54e93e4de492d3444721478ead8db085caed788ac8dfe8b2a93fe454e6"
Nov 03 20:54:25.889025 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.888985    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"]
Nov 03 20:54:25.896188 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.896163    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.983629 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.983592    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"]
Nov 03 20:54:25.988151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.987825    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.988151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.987955    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.988151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.988004    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a185b07-d07c-4a15-ba9d-adddb724d421-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.988151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.988051    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m66x4\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-kube-api-access-m66x4\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.988151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.988132    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.988542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.988171    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.988542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.988279    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.988542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.988360    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:25.988542 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:25.988407    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090060 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090016    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090139    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090179    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a185b07-d07c-4a15-ba9d-adddb724d421-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090226    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m66x4\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-kube-api-access-m66x4\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090296    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090477 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090340    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090487    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090524    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-certs\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090592 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090568    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090686    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090715    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-data\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.090763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090751    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.091006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090983    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a185b07-d07c-4a15-ba9d-adddb724d421-istiod-ca-cert\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.091006 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.090994    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-credential-socket\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.093021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.092996    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-envoy\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.093398 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.093378    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-podinfo\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.148450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.148377    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-token\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.148601 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.148535    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m66x4\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-kube-api-access-m66x4\") pod \"e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") " pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.213236 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.213197    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:26.656289 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.656249    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"]
Nov 03 20:54:26.657971 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:54:26.657942    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a185b07_d07c_4a15_ba9d_adddb724d421.slice/crio-b681f963dc92ea65cae0a1fb1d2f89a3816df9a784beec0c6965872891a7ec19 WatchSource:0}: Error finding container b681f963dc92ea65cae0a1fb1d2f89a3816df9a784beec0c6965872891a7ec19: Status 404 returned error can't find the container with id b681f963dc92ea65cae0a1fb1d2f89a3816df9a784beec0c6965872891a7ec19
Nov 03 20:54:26.661013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.660973    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:54:26.661136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.661068    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:54:26.661183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:26.661137    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:54:27.057443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:27.057348    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" event={"ID":"5a185b07-d07c-4a15-ba9d-adddb724d421","Type":"ContainerStarted","Data":"b0de665c74aa3eb99d57f3801e01611b57927786bcea71b4f72d54332c32fbcc"}
Nov 03 20:54:27.057443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:27.057405    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" event={"ID":"5a185b07-d07c-4a15-ba9d-adddb724d421","Type":"ContainerStarted","Data":"b681f963dc92ea65cae0a1fb1d2f89a3816df9a784beec0c6965872891a7ec19"}
Nov 03 20:54:27.215271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:27.215112    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:27.228247 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:27.228209    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 20:54:27.228453 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:27.228263    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 20:54:27.278995 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:27.278929    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podStartSLOduration=2.27890719 podStartE2EDuration="2.27890719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:27.228122547 +0000 UTC m=+1507.232081276" watchObservedRunningTime="2026-11-03 20:54:27.27890719 +0000 UTC m=+1507.282865920"
Nov 03 20:54:27.561540 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:27.561499    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:28.213942 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:28.213904    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 20:54:28.214418 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:28.213985    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 20:54:29.213825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:29.213784    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 20:54:29.214013 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:29.213865    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 20:54:30.213685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:30.213646    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 20:54:30.213887 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:30.213714    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 20:54:31.214434 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:31.214399    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 20:54:31.214834 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:31.214464    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 20:54:32.214180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:32.214138    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 20:54:32.214342 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:32.214216    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 20:54:32.367223 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:32.367187    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:33.214294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:33.214262    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 03 20:54:33.214485 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:33.214322    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 03 20:54:34.219211 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:34.219180    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:34.219658 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:34.219645    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:34.220474 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:34.220451    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 20:54:36.807224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:36.807173    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:41.437159 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.436888    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"]
Nov 03 20:54:41.446232 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.446201    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.491460 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.491423    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"]
Nov 03 20:54:41.531411 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.531374    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:54:41.531411 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.531394    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:54:41.577235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.577194    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.577424 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.577352    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbs67\" (UniqueName: \"kubernetes.io/projected/3b4a4be2-5381-4913-b42e-a90d1f950ad9-kube-api-access-hbs67\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.577515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.577499    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.679391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.679351    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.679580 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.679448    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbs67\" (UniqueName: \"kubernetes.io/projected/3b4a4be2-5381-4913-b42e-a90d1f950ad9-kube-api-access-hbs67\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.679580 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.679550    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.680184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.680159    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.682179 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.682158    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.733435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.733360    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbs67\" (UniqueName: \"kubernetes.io/projected/3b4a4be2-5381-4913-b42e-a90d1f950ad9-kube-api-access-hbs67\") pod \"maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.765636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.765601    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:41.907837 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:41.907789    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:42.157560 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:42.157531    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"]
Nov 03 20:54:42.158339 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:54:42.158311    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4a4be2_5381_4913_b42e_a90d1f950ad9.slice/crio-f4798212428931787e90b044e630499f4a562a95bac426076f4d538d7051e4a6 WatchSource:0}: Error finding container f4798212428931787e90b044e630499f4a562a95bac426076f4d538d7051e4a6: Status 404 returned error can't find the container with id f4798212428931787e90b044e630499f4a562a95bac426076f4d538d7051e4a6
Nov 03 20:54:42.235650 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:42.235615    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl" event={"ID":"3b4a4be2-5381-4913-b42e-a90d1f950ad9","Type":"ContainerStarted","Data":"f4798212428931787e90b044e630499f4a562a95bac426076f4d538d7051e4a6"}
Nov 03 20:54:43.249081 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:43.249037    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl" event={"ID":"3b4a4be2-5381-4913-b42e-a90d1f950ad9","Type":"ContainerStarted","Data":"95cea79ab72c62a50537985a62a95af90539539e8ab82744a97a73a9eb733336"}
Nov 03 20:54:43.357548 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:43.357498    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl" podStartSLOduration=2.357484764 podStartE2EDuration="2.357484764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:43.356796224 +0000 UTC m=+1523.360754969" watchObservedRunningTime="2026-11-03 20:54:43.357484764 +0000 UTC m=+1523.361443487"
Nov 03 20:54:44.261691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:44.261653    5196 generic.go:356] "Generic (PLEG): container finished" podID="3b4a4be2-5381-4913-b42e-a90d1f950ad9" containerID="95cea79ab72c62a50537985a62a95af90539539e8ab82744a97a73a9eb733336" exitCode=0
Nov 03 20:54:44.262187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:44.261735    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl" event={"ID":"3b4a4be2-5381-4913-b42e-a90d1f950ad9","Type":"ContainerDied","Data":"95cea79ab72c62a50537985a62a95af90539539e8ab82744a97a73a9eb733336"}
Nov 03 20:54:44.446416 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:44.446377    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:45.481113 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.481068    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:45.540902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.540794    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4a4be2-5381-4913-b42e-a90d1f950ad9-kube-api-access-hbs67\" (UniqueName: \"kubernetes.io/projected/3b4a4be2-5381-4913-b42e-a90d1f950ad9-kube-api-access-hbs67\") pod \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") "
Nov 03 20:54:45.541072 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.541043    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-openshift-ca\") pod \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") "
Nov 03 20:54:45.541215 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.541159    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-xks-ca\") pod \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\" (UID: \"3b4a4be2-5381-4913-b42e-a90d1f950ad9\") "
Nov 03 20:54:45.541468 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.541437    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-openshift-ca" pod "3b4a4be2-5381-4913-b42e-a90d1f950ad9" (UID: "3b4a4be2-5381-4913-b42e-a90d1f950ad9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:45.542032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.542011    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:45.545153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.543705    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-xks-ca" pod "3b4a4be2-5381-4913-b42e-a90d1f950ad9" (UID: "3b4a4be2-5381-4913-b42e-a90d1f950ad9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:54:45.545153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.543728    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4a4be2-5381-4913-b42e-a90d1f950ad9-kube-api-access-hbs67" pod "3b4a4be2-5381-4913-b42e-a90d1f950ad9" (UID: "3b4a4be2-5381-4913-b42e-a90d1f950ad9"). InnerVolumeSpecName "kube-api-access-hbs67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:45.643590 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.643548    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4a4be2-5381-4913-b42e-a90d1f950ad9-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:45.643590 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:45.643586    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbs67\" (UniqueName: \"kubernetes.io/projected/3b4a4be2-5381-4913-b42e-a90d1f950ad9-kube-api-access-hbs67\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:46.124399 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:46.124355    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:54:46.285838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:46.285807    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"
Nov 03 20:54:46.285838 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:46.285834    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl" event={"ID":"3b4a4be2-5381-4913-b42e-a90d1f950ad9","Type":"ContainerDied","Data":"f4798212428931787e90b044e630499f4a562a95bac426076f4d538d7051e4a6"}
Nov 03 20:54:46.286051 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:46.285888    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4798212428931787e90b044e630499f4a562a95bac426076f4d538d7051e4a6"
Nov 03 20:54:47.179568 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:47.175586    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:48.488266 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:54:48.488230    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:54:52.497747 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:52.497700    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:54.113950 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:54.113855    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"]
Nov 03 20:54:54.189125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:54.189072    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"]
Nov 03 20:54:54.453497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:54.453404    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:54.643499 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:54.643462    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:57.478687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.478651    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"]
Nov 03 20:54:57.479565 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.479548    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3b4a4be2-5381-4913-b42e-a90d1f950ad9" containerName="revoke-keys"
Nov 03 20:54:57.479565 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.479565    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4a4be2-5381-4913-b42e-a90d1f950ad9" containerName="revoke-keys"
Nov 03 20:54:57.479739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.479729    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4a4be2-5381-4913-b42e-a90d1f950ad9" containerName="revoke-keys"
Nov 03 20:54:57.484331 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.484306    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.495081 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.495051    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.495271 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.495213    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxhp\" (UniqueName: \"kubernetes.io/projected/e56b34a8-2a7e-406f-9596-56bf6651b982-kube-api-access-cpxhp\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.495340 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.495318    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.535388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.535348    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"]
Nov 03 20:54:57.574335 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.574291    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:54:57.574488 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.574442    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:54:57.597841 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.597801    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.598067 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.597879    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxhp\" (UniqueName: \"kubernetes.io/projected/e56b34a8-2a7e-406f-9596-56bf6651b982-kube-api-access-cpxhp\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.598067 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.597986    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.599011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.598985    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.601373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.601339    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.652633 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.652593    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxhp\" (UniqueName: \"kubernetes.io/projected/e56b34a8-2a7e-406f-9596-56bf6651b982-kube-api-access-cpxhp\") pod \"maas-api-revoke-keys-e2e-ait-43693d42-7vqg5\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:57.672220 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.672070    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:57.802028 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:57.801933    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:54:58.206302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:58.206276    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"]
Nov 03 20:54:58.208743 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:54:58.208693    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56b34a8_2a7e_406f_9596_56bf6651b982.slice/crio-fa1ae0edef1b270af2df78a52ba454fe409845fbef2a92375de1a29aee8de59f WatchSource:0}: Error finding container fa1ae0edef1b270af2df78a52ba454fe409845fbef2a92375de1a29aee8de59f: Status 404 returned error can't find the container with id fa1ae0edef1b270af2df78a52ba454fe409845fbef2a92375de1a29aee8de59f
Nov 03 20:54:58.411883 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:58.411851    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5" event={"ID":"e56b34a8-2a7e-406f-9596-56bf6651b982","Type":"ContainerStarted","Data":"fa1ae0edef1b270af2df78a52ba454fe409845fbef2a92375de1a29aee8de59f"}
Nov 03 20:54:59.115238 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.115197    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg" podUID="5f2c0699-760a-4073-99c2-903788909dba" containerName="payload-processing" containerID="cri-o://d44bc83ac964f190efa83c2ef27fb6de5f85a8d9dcf7c097c415981c906522cb" gracePeriod=25
Nov 03 20:54:59.190723 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.190673    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd" podUID="49cbe6b2-ff91-490d-9491-6cdcb8e201e5" containerName="payload-pre-processing" containerID="cri-o://1fec91e4a26d4313a1720a42f8e8d6e229af6ef28320d445563abadfb9e2b9c5" gracePeriod=25
Nov 03 20:54:59.432386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.432329    5196 generic.go:356] "Generic (PLEG): container finished" podID="49cbe6b2-ff91-490d-9491-6cdcb8e201e5" containerID="1fec91e4a26d4313a1720a42f8e8d6e229af6ef28320d445563abadfb9e2b9c5" exitCode=0
Nov 03 20:54:59.432647 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.432412    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd" event={"ID":"49cbe6b2-ff91-490d-9491-6cdcb8e201e5","Type":"ContainerDied","Data":"1fec91e4a26d4313a1720a42f8e8d6e229af6ef28320d445563abadfb9e2b9c5"}
Nov 03 20:54:59.435257 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.435224    5196 generic.go:356] "Generic (PLEG): container finished" podID="5f2c0699-760a-4073-99c2-903788909dba" containerID="d44bc83ac964f190efa83c2ef27fb6de5f85a8d9dcf7c097c415981c906522cb" exitCode=0
Nov 03 20:54:59.435443 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.435364    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg" event={"ID":"5f2c0699-760a-4073-99c2-903788909dba","Type":"ContainerDied","Data":"d44bc83ac964f190efa83c2ef27fb6de5f85a8d9dcf7c097c415981c906522cb"}
Nov 03 20:54:59.438685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.438648    5196 generic.go:356] "Generic (PLEG): container finished" podID="e56b34a8-2a7e-406f-9596-56bf6651b982" containerID="a0191b3e9efd97958290389b47343c539eed27625523df5613d9e1977cde99cf" exitCode=0
Nov 03 20:54:59.440032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.439997    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5" event={"ID":"e56b34a8-2a7e-406f-9596-56bf6651b982","Type":"ContainerDied","Data":"a0191b3e9efd97958290389b47343c539eed27625523df5613d9e1977cde99cf"}
Nov 03 20:54:59.488349 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.488314    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:54:59.521440 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.521412    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:54:59.533146 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.533079    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-kube-api-access-hhc4d\" (UniqueName: \"kubernetes.io/projected/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-kube-api-access-hhc4d\") pod \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\" (UID: \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\") "
Nov 03 20:54:59.533354 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.533233    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-plugins-config-volume\") pod \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\" (UID: \"49cbe6b2-ff91-490d-9491-6cdcb8e201e5\") "
Nov 03 20:54:59.533354 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.533273    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2c0699-760a-4073-99c2-903788909dba-kube-api-access-6j24m\" (UniqueName: \"kubernetes.io/projected/5f2c0699-760a-4073-99c2-903788909dba-kube-api-access-6j24m\") pod \"5f2c0699-760a-4073-99c2-903788909dba\" (UID: \"5f2c0699-760a-4073-99c2-903788909dba\") "
Nov 03 20:54:59.533489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.533389    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2c0699-760a-4073-99c2-903788909dba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c0699-760a-4073-99c2-903788909dba-plugins-config-volume\") pod \"5f2c0699-760a-4073-99c2-903788909dba\" (UID: \"5f2c0699-760a-4073-99c2-903788909dba\") "
Nov 03 20:54:59.534052 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.534022    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2c0699-760a-4073-99c2-903788909dba-plugins-config-volume" pod "5f2c0699-760a-4073-99c2-903788909dba" (UID: "5f2c0699-760a-4073-99c2-903788909dba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:59.534052 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.534038    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-plugins-config-volume" pod "49cbe6b2-ff91-490d-9491-6cdcb8e201e5" (UID: "49cbe6b2-ff91-490d-9491-6cdcb8e201e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:59.534536 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.534510    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c0699-760a-4073-99c2-903788909dba-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:59.534614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.534542    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:59.536291 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.536263    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-kube-api-access-hhc4d" pod "49cbe6b2-ff91-490d-9491-6cdcb8e201e5" (UID: "49cbe6b2-ff91-490d-9491-6cdcb8e201e5"). InnerVolumeSpecName "kube-api-access-hhc4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:59.536567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.536539    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2c0699-760a-4073-99c2-903788909dba-kube-api-access-6j24m" pod "5f2c0699-760a-4073-99c2-903788909dba" (UID: "5f2c0699-760a-4073-99c2-903788909dba"). InnerVolumeSpecName "kube-api-access-6j24m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:59.636214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.636111    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhc4d\" (UniqueName: \"kubernetes.io/projected/49cbe6b2-ff91-490d-9491-6cdcb8e201e5-kube-api-access-hhc4d\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:59.636214 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:54:59.636143    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j24m\" (UniqueName: \"kubernetes.io/projected/5f2c0699-760a-4073-99c2-903788909dba-kube-api-access-6j24m\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:00.451134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.451033    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"
Nov 03 20:55:00.451134 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.451060    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd" event={"ID":"49cbe6b2-ff91-490d-9491-6cdcb8e201e5","Type":"ContainerDied","Data":"b5b2990c26e7f9cacd9e5553430f610b6a6fd975282acb1dc7805ebe8e6fb86f"}
Nov 03 20:55:00.451679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.451148    5196 scope.go:122] "RemoveContainer" containerID="1fec91e4a26d4313a1720a42f8e8d6e229af6ef28320d445563abadfb9e2b9c5"
Nov 03 20:55:00.452933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.452911    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"
Nov 03 20:55:00.453062 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.452933    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg" event={"ID":"5f2c0699-760a-4073-99c2-903788909dba","Type":"ContainerDied","Data":"8d56cddfd05bb441dbd357392c2e5b4fa9707be534ebe2660dd1ea802c6405c7"}
Nov 03 20:55:00.469803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.469772    5196 scope.go:122] "RemoveContainer" containerID="d44bc83ac964f190efa83c2ef27fb6de5f85a8d9dcf7c097c415981c906522cb"
Nov 03 20:55:00.495163 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.495111    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd" podUID="49cbe6b2-ff91-490d-9491-6cdcb8e201e5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.168:50052\" within 1s: context deadline exceeded"
Nov 03 20:55:00.495344 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:55:00.495232    5196 logging.go:55] [core] [Channel #3743 SubChannel #3744]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.168:50052", ServerName: "10.132.0.168:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.168:50052: operation was canceled"
Nov 03 20:55:00.613495 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.613462    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"]
Nov 03 20:55:00.625777 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.625621    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e20715ef-847876c7c5-rxzgg"]
Nov 03 20:55:00.657576 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.657543    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:55:00.718476 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.718385    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"]
Nov 03 20:55:00.724793 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.724761    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e20715ef-588bdbc7c5-xhhfd"]
Nov 03 20:55:00.754714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.754674    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56b34a8-2a7e-406f-9596-56bf6651b982-kube-api-access-cpxhp\" (UniqueName: \"kubernetes.io/projected/e56b34a8-2a7e-406f-9596-56bf6651b982-kube-api-access-cpxhp\") pod \"e56b34a8-2a7e-406f-9596-56bf6651b982\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") "
Nov 03 20:55:00.754714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.754715    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-openshift-ca\") pod \"e56b34a8-2a7e-406f-9596-56bf6651b982\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") "
Nov 03 20:55:00.754928 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.754859    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-xks-ca\") pod \"e56b34a8-2a7e-406f-9596-56bf6651b982\" (UID: \"e56b34a8-2a7e-406f-9596-56bf6651b982\") "
Nov 03 20:55:00.755155 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.755134    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-openshift-ca" pod "e56b34a8-2a7e-406f-9596-56bf6651b982" (UID: "e56b34a8-2a7e-406f-9596-56bf6651b982"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:00.755581 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.755564    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:00.757173 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.757148    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-xks-ca" pod "e56b34a8-2a7e-406f-9596-56bf6651b982" (UID: "e56b34a8-2a7e-406f-9596-56bf6651b982"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:00.757266 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.757239    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56b34a8-2a7e-406f-9596-56bf6651b982-kube-api-access-cpxhp" pod "e56b34a8-2a7e-406f-9596-56bf6651b982" (UID: "e56b34a8-2a7e-406f-9596-56bf6651b982"). InnerVolumeSpecName "kube-api-access-cpxhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:00.857136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.857075    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56b34a8-2a7e-406f-9596-56bf6651b982-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:00.857136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:00.857139    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxhp\" (UniqueName: \"kubernetes.io/projected/e56b34a8-2a7e-406f-9596-56bf6651b982-kube-api-access-cpxhp\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:01.469852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:01.469817    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"
Nov 03 20:55:01.469852 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:01.469839    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5" event={"ID":"e56b34a8-2a7e-406f-9596-56bf6651b982","Type":"ContainerDied","Data":"fa1ae0edef1b270af2df78a52ba454fe409845fbef2a92375de1a29aee8de59f"}
Nov 03 20:55:01.470340 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:01.469887    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa1ae0edef1b270af2df78a52ba454fe409845fbef2a92375de1a29aee8de59f"
Nov 03 20:55:02.112546 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:02.112503    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cbe6b2-ff91-490d-9491-6cdcb8e201e5" path="/var/lib/kubelet/pods/49cbe6b2-ff91-490d-9491-6cdcb8e201e5/volumes"
Nov 03 20:55:02.113160 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:02.113137    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2c0699-760a-4073-99c2-903788909dba" path="/var/lib/kubelet/pods/5f2c0699-760a-4073-99c2-903788909dba/volumes"
Nov 03 20:55:02.873107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:02.873057    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:07.855691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:07.855653    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:09.461825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:09.461780    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:09.615076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:09.615041    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:12.005140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.004627    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:12.162638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.162592    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:12.593426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.593388    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"]
Nov 03 20:55:12.594262 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594246    5196 cpu_manager.go:414] "containerMap: removing container" podUID="49cbe6b2-ff91-490d-9491-6cdcb8e201e5" containerName="payload-pre-processing"
Nov 03 20:55:12.594348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594263    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cbe6b2-ff91-490d-9491-6cdcb8e201e5" containerName="payload-pre-processing"
Nov 03 20:55:12.594348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594278    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e56b34a8-2a7e-406f-9596-56bf6651b982" containerName="revoke-keys"
Nov 03 20:55:12.594348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594283    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56b34a8-2a7e-406f-9596-56bf6651b982" containerName="revoke-keys"
Nov 03 20:55:12.594348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594308    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5f2c0699-760a-4073-99c2-903788909dba" containerName="payload-processing"
Nov 03 20:55:12.594348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594313    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2c0699-760a-4073-99c2-903788909dba" containerName="payload-processing"
Nov 03 20:55:12.594500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594423    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2c0699-760a-4073-99c2-903788909dba" containerName="payload-processing"
Nov 03 20:55:12.594500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594438    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cbe6b2-ff91-490d-9491-6cdcb8e201e5" containerName="payload-pre-processing"
Nov 03 20:55:12.594500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.594444    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56b34a8-2a7e-406f-9596-56bf6651b982" containerName="revoke-keys"
Nov 03 20:55:12.598723 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.598693    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.648338 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.648304    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:55:12.652198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.652165    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"]
Nov 03 20:55:12.691056 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.691017    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:55:12.761087 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.761050    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:12.773537 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.773503    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.773709 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.773654    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8nct\" (UniqueName: \"kubernetes.io/projected/34ff596d-0781-4bce-b686-15442140f7dd-kube-api-access-b8nct\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.773797 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.773782    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.876066 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.875977    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.876277 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.876143    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8nct\" (UniqueName: \"kubernetes.io/projected/34ff596d-0781-4bce-b686-15442140f7dd-kube-api-access-b8nct\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.876358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.876282    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.877587 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.877551    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.879842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.879812    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:12.931189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:12.931148    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8nct\" (UniqueName: \"kubernetes.io/projected/34ff596d-0781-4bce-b686-15442140f7dd-kube-api-access-b8nct\") pod \"maas-api-revoke-sub-64b45d03-r9m7h\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:13.215282 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:13.215205    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:13.605685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:13.605658    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"]
Nov 03 20:55:13.607981 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:55:13.607937    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ff596d_0781_4bce_b686_15442140f7dd.slice/crio-0bd3d40fc1fa67f40c3df1c1e7678b9ba51750f89a28a69b3aac27ce0ce52b32 WatchSource:0}: Error finding container 0bd3d40fc1fa67f40c3df1c1e7678b9ba51750f89a28a69b3aac27ce0ce52b32: Status 404 returned error can't find the container with id 0bd3d40fc1fa67f40c3df1c1e7678b9ba51750f89a28a69b3aac27ce0ce52b32
Nov 03 20:55:13.650756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:13.650721    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h" event={"ID":"34ff596d-0781-4bce-b686-15442140f7dd","Type":"ContainerStarted","Data":"0bd3d40fc1fa67f40c3df1c1e7678b9ba51750f89a28a69b3aac27ce0ce52b32"}
Nov 03 20:55:14.048687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:14.048655    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"]
Nov 03 20:55:14.094457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:14.094422    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b08f1e87-4jmmn"]
Nov 03 20:55:14.111073 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:14.111039    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b32de33-fd09-4b7d-9753-021355e3f33d" path="/var/lib/kubelet/pods/5b32de33-fd09-4b7d-9753-021355e3f33d/volumes"
Nov 03 20:55:14.662815 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:14.662782    5196 generic.go:356] "Generic (PLEG): container finished" podID="34ff596d-0781-4bce-b686-15442140f7dd" containerID="fb8ed6e0d4ac84b195e318a828680ecba7ac2e2c7bda4fe090af98dd1fe77fbe" exitCode=0
Nov 03 20:55:14.663207 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:14.662866    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h" event={"ID":"34ff596d-0781-4bce-b686-15442140f7dd","Type":"ContainerDied","Data":"fb8ed6e0d4ac84b195e318a828680ecba7ac2e2c7bda4fe090af98dd1fe77fbe"}
Nov 03 20:55:15.861715 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:15.861684    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:15.928575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:15.928493    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-openshift-ca\") pod \"34ff596d-0781-4bce-b686-15442140f7dd\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") "
Nov 03 20:55:15.928749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:15.928623    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-xks-ca\") pod \"34ff596d-0781-4bce-b686-15442140f7dd\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") "
Nov 03 20:55:15.928905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:15.928871    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ff596d-0781-4bce-b686-15442140f7dd-kube-api-access-b8nct\" (UniqueName: \"kubernetes.io/projected/34ff596d-0781-4bce-b686-15442140f7dd-kube-api-access-b8nct\") pod \"34ff596d-0781-4bce-b686-15442140f7dd\" (UID: \"34ff596d-0781-4bce-b686-15442140f7dd\") "
Nov 03 20:55:15.928905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:15.928891    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-openshift-ca" pod "34ff596d-0781-4bce-b686-15442140f7dd" (UID: "34ff596d-0781-4bce-b686-15442140f7dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:15.929827 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:15.929802    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:15.931186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:15.931159    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-xks-ca" pod "34ff596d-0781-4bce-b686-15442140f7dd" (UID: "34ff596d-0781-4bce-b686-15442140f7dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:15.931302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:15.931255    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ff596d-0781-4bce-b686-15442140f7dd-kube-api-access-b8nct" pod "34ff596d-0781-4bce-b686-15442140f7dd" (UID: "34ff596d-0781-4bce-b686-15442140f7dd"). InnerVolumeSpecName "kube-api-access-b8nct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:16.032330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:16.032293    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ff596d-0781-4bce-b686-15442140f7dd-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:16.032330 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:16.032331    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8nct\" (UniqueName: \"kubernetes.io/projected/34ff596d-0781-4bce-b686-15442140f7dd-kube-api-access-b8nct\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:16.687340 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:16.687312    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"
Nov 03 20:55:16.687545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:16.687342    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h" event={"ID":"34ff596d-0781-4bce-b686-15442140f7dd","Type":"ContainerDied","Data":"0bd3d40fc1fa67f40c3df1c1e7678b9ba51750f89a28a69b3aac27ce0ce52b32"}
Nov 03 20:55:16.687545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:16.687423    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd3d40fc1fa67f40c3df1c1e7678b9ba51750f89a28a69b3aac27ce0ce52b32"
Nov 03 20:55:17.779602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:17.779569    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:18.075582 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.075545    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"]
Nov 03 20:55:18.076426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.076409    5196 cpu_manager.go:414] "containerMap: removing container" podUID="34ff596d-0781-4bce-b686-15442140f7dd" containerName="revoke-keys"
Nov 03 20:55:18.076426 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.076426    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ff596d-0781-4bce-b686-15442140f7dd" containerName="revoke-keys"
Nov 03 20:55:18.076633 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.076621    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ff596d-0781-4bce-b686-15442140f7dd" containerName="revoke-keys"
Nov 03 20:55:18.081575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.081555    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.132877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.132847    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"]
Nov 03 20:55:18.172351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.172319    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:55:18.172581 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.172394    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:55:18.179881 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.179844    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r89hh\" (UniqueName: \"kubernetes.io/projected/b6504485-a679-49ad-8633-74677c3b5205-kube-api-access-r89hh\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.180301 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.180280    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6504485-a679-49ad-8633-74677c3b5205-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.180444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.180305    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6504485-a679-49ad-8633-74677c3b5205-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.281996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.281958    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6504485-a679-49ad-8633-74677c3b5205-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.281996 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.281996    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6504485-a679-49ad-8633-74677c3b5205-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.282246 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.282130    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r89hh\" (UniqueName: \"kubernetes.io/projected/b6504485-a679-49ad-8633-74677c3b5205-kube-api-access-r89hh\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.282798 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.282775    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6504485-a679-49ad-8633-74677c3b5205-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.284684 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.284658    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6504485-a679-49ad-8633-74677c3b5205-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.336971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.336894    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r89hh\" (UniqueName: \"kubernetes.io/projected/b6504485-a679-49ad-8633-74677c3b5205-kube-api-access-r89hh\") pod \"maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.396638 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.396597    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:18.497648 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:55:18.497581    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:55:18.784437 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:18.784398    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"]
Nov 03 20:55:18.786624 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:55:18.786596    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6504485_a679_49ad_8633_74677c3b5205.slice/crio-7dad7443cea92d79738231c07ff6de3d74c8cee429cfa8e85bb16f7fe831f217 WatchSource:0}: Error finding container 7dad7443cea92d79738231c07ff6de3d74c8cee429cfa8e85bb16f7fe831f217: Status 404 returned error can't find the container with id 7dad7443cea92d79738231c07ff6de3d74c8cee429cfa8e85bb16f7fe831f217
Nov 03 20:55:19.718377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:19.718340    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89" event={"ID":"b6504485-a679-49ad-8633-74677c3b5205","Type":"ContainerStarted","Data":"81233f0733369ad39a81b026107804379358f564f6642c0b0cc441c9cc4028bb"}
Nov 03 20:55:19.718554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:19.718438    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89" event={"ID":"b6504485-a679-49ad-8633-74677c3b5205","Type":"ContainerStarted","Data":"7dad7443cea92d79738231c07ff6de3d74c8cee429cfa8e85bb16f7fe831f217"}
Nov 03 20:55:19.831248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:19.831190    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89" podStartSLOduration=1.8311770520000001 podStartE2EDuration="1.831177052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:19.829233121 +0000 UTC m=+1559.833191847" watchObservedRunningTime="2026-11-03 20:55:19.831177052 +0000 UTC m=+1559.835135775"
Nov 03 20:55:20.729807 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:20.729767    5196 generic.go:356] "Generic (PLEG): container finished" podID="b6504485-a679-49ad-8633-74677c3b5205" containerID="81233f0733369ad39a81b026107804379358f564f6642c0b0cc441c9cc4028bb" exitCode=0
Nov 03 20:55:20.730021 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:20.729846    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89" event={"ID":"b6504485-a679-49ad-8633-74677c3b5205","Type":"ContainerDied","Data":"81233f0733369ad39a81b026107804379358f564f6642c0b0cc441c9cc4028bb"}
Nov 03 20:55:21.942756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:21.942726    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:22.051821 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.051722    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6504485-a679-49ad-8633-74677c3b5205-kube-api-access-r89hh\" (UniqueName: \"kubernetes.io/projected/b6504485-a679-49ad-8633-74677c3b5205-kube-api-access-r89hh\") pod \"b6504485-a679-49ad-8633-74677c3b5205\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") "
Nov 03 20:55:22.052301 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.052277    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6504485-a679-49ad-8633-74677c3b5205-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6504485-a679-49ad-8633-74677c3b5205-maas-api-openshift-ca\") pod \"b6504485-a679-49ad-8633-74677c3b5205\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") "
Nov 03 20:55:22.052527 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.052512    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6504485-a679-49ad-8633-74677c3b5205-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6504485-a679-49ad-8633-74677c3b5205-maas-api-xks-ca\") pod \"b6504485-a679-49ad-8633-74677c3b5205\" (UID: \"b6504485-a679-49ad-8633-74677c3b5205\") "
Nov 03 20:55:22.068525 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.068484    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6504485-a679-49ad-8633-74677c3b5205-maas-api-openshift-ca" pod "b6504485-a679-49ad-8633-74677c3b5205" (UID: "b6504485-a679-49ad-8633-74677c3b5205"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:22.069613 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.069582    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6504485-a679-49ad-8633-74677c3b5205-maas-api-xks-ca" pod "b6504485-a679-49ad-8633-74677c3b5205" (UID: "b6504485-a679-49ad-8633-74677c3b5205"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:22.115620 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.115587    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6504485-a679-49ad-8633-74677c3b5205-kube-api-access-r89hh" pod "b6504485-a679-49ad-8633-74677c3b5205" (UID: "b6504485-a679-49ad-8633-74677c3b5205"). InnerVolumeSpecName "kube-api-access-r89hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:22.164187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.160552    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r89hh\" (UniqueName: \"kubernetes.io/projected/b6504485-a679-49ad-8633-74677c3b5205-kube-api-access-r89hh\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:22.164187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.160593    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6504485-a679-49ad-8633-74677c3b5205-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:22.164187 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.160612    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6504485-a679-49ad-8633-74677c3b5205-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:22.754450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.754412    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"
Nov 03 20:55:22.754450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.754431    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89" event={"ID":"b6504485-a679-49ad-8633-74677c3b5205","Type":"ContainerDied","Data":"7dad7443cea92d79738231c07ff6de3d74c8cee429cfa8e85bb16f7fe831f217"}
Nov 03 20:55:22.754668 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.754477    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dad7443cea92d79738231c07ff6de3d74c8cee429cfa8e85bb16f7fe831f217"
Nov 03 20:55:22.950016 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:22.949977    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:23.221812 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:23.221775    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:24.058435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:24.058400    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"]
Nov 03 20:55:24.111876 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:24.111843    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5929db-nlzgd"]
Nov 03 20:55:24.119458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:24.119424    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:24.950104 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:24.949995    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:25.151310 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:25.151270    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:25.563073 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:25.563041    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:26.113309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:26.113266    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b37974d-46c9-42a9-96bc-1e8bd9f8f83a" path="/var/lib/kubelet/pods/3b37974d-46c9-42a9-96bc-1e8bd9f8f83a/volumes"
Nov 03 20:55:28.094252 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:28.094212    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:28.725082 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:28.725046    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:29.003920 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:29.003830    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:29.662256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:29.662219    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:29.812288 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:29.812253    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:31.892136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:31.892084    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:32.057756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.057710    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:32.582265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.582230    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:32.729898 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.729852    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"]
Nov 03 20:55:32.730803 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.730783    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b6504485-a679-49ad-8633-74677c3b5205" containerName="revoke-keys"
Nov 03 20:55:32.730874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.730805    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6504485-a679-49ad-8633-74677c3b5205" containerName="revoke-keys"
Nov 03 20:55:32.730968 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.730958    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6504485-a679-49ad-8633-74677c3b5205" containerName="revoke-keys"
Nov 03 20:55:32.735301 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.735275    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:32.808637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.808601    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"]
Nov 03 20:55:32.829830 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.829795    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:55:32.830118 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.829865    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:55:32.884276 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.884183    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:32.884450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.884305    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6r2t\" (UniqueName: \"kubernetes.io/projected/b5cb8b8c-6a6c-4c23-8761-002d3a831053-kube-api-access-v6r2t\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:32.884450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.884377    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:32.985918 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.985879    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:32.986383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.986081    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:32.986383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.986206    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6r2t\" (UniqueName: \"kubernetes.io/projected/b5cb8b8c-6a6c-4c23-8761-002d3a831053-kube-api-access-v6r2t\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:32.986885 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.986860    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:32.988828 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:32.988799    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:33.041031 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:33.040994    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6r2t\" (UniqueName: \"kubernetes.io/projected/b5cb8b8c-6a6c-4c23-8761-002d3a831053-kube-api-access-v6r2t\") pod \"maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:33.080431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:33.080391    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:33.483577 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:33.483546    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"]
Nov 03 20:55:33.485397 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:55:33.485369    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cb8b8c_6a6c_4c23_8761_002d3a831053.slice/crio-cf624b2695b5cc96475e1c614b40b6391f1ee45ed14c695e96452b4f39e4dbd9 WatchSource:0}: Error finding container cf624b2695b5cc96475e1c614b40b6391f1ee45ed14c695e96452b4f39e4dbd9: Status 404 returned error can't find the container with id cf624b2695b5cc96475e1c614b40b6391f1ee45ed14c695e96452b4f39e4dbd9
Nov 03 20:55:33.897877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:33.897826    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2" event={"ID":"b5cb8b8c-6a6c-4c23-8761-002d3a831053","Type":"ContainerStarted","Data":"cf624b2695b5cc96475e1c614b40b6391f1ee45ed14c695e96452b4f39e4dbd9"}
Nov 03 20:55:34.188031 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.187990    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:34.460726 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.460624    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:34.618216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.618180    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:34.774244 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.774189    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:34.783014 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.782978    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"]
Nov 03 20:55:34.840991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.840959    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e20715ef-q6qnl"]
Nov 03 20:55:34.923832 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.923795    5196 generic.go:356] "Generic (PLEG): container finished" podID="b5cb8b8c-6a6c-4c23-8761-002d3a831053" containerID="8c89741cd40489a7a8e0132b2b4b76aab2bd489cd8501af28d3531bea98e21ad" exitCode=0
Nov 03 20:55:34.924034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.923952    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2" event={"ID":"b5cb8b8c-6a6c-4c23-8761-002d3a831053","Type":"ContainerDied","Data":"8c89741cd40489a7a8e0132b2b4b76aab2bd489cd8501af28d3531bea98e21ad"}
Nov 03 20:55:34.927227 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:34.926917    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:35.501184 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:35.501145    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:36.172140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.171252    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4a4be2-5381-4913-b42e-a90d1f950ad9" path="/var/lib/kubelet/pods/3b4a4be2-5381-4913-b42e-a90d1f950ad9/volumes"
Nov 03 20:55:36.342497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.342348    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:36.389823 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.389744    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-xks-ca\") pod \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") "
Nov 03 20:55:36.390378 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.390351    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cb8b8c-6a6c-4c23-8761-002d3a831053-kube-api-access-v6r2t\" (UniqueName: \"kubernetes.io/projected/b5cb8b8c-6a6c-4c23-8761-002d3a831053-kube-api-access-v6r2t\") pod \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") "
Nov 03 20:55:36.390574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.390558    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-openshift-ca\") pod \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\" (UID: \"b5cb8b8c-6a6c-4c23-8761-002d3a831053\") "
Nov 03 20:55:36.391611 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.391208    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-openshift-ca" pod "b5cb8b8c-6a6c-4c23-8761-002d3a831053" (UID: "b5cb8b8c-6a6c-4c23-8761-002d3a831053"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:36.391611 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.391454    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:36.392387 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.392358    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-xks-ca" pod "b5cb8b8c-6a6c-4c23-8761-002d3a831053" (UID: "b5cb8b8c-6a6c-4c23-8761-002d3a831053"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:36.392786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.392769    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cb8b8c-6a6c-4c23-8761-002d3a831053-kube-api-access-v6r2t" pod "b5cb8b8c-6a6c-4c23-8761-002d3a831053" (UID: "b5cb8b8c-6a6c-4c23-8761-002d3a831053"). InnerVolumeSpecName "kube-api-access-v6r2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:36.493877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.493830    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6r2t\" (UniqueName: \"kubernetes.io/projected/b5cb8b8c-6a6c-4c23-8761-002d3a831053-kube-api-access-v6r2t\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:36.493877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.493880    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cb8b8c-6a6c-4c23-8761-002d3a831053-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:36.541050 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.540951    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:36.751435 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.751403    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:36.984083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.984046    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"
Nov 03 20:55:36.984083 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.984071    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2" event={"ID":"b5cb8b8c-6a6c-4c23-8761-002d3a831053","Type":"ContainerDied","Data":"cf624b2695b5cc96475e1c614b40b6391f1ee45ed14c695e96452b4f39e4dbd9"}
Nov 03 20:55:36.984365 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:36.984154    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf624b2695b5cc96475e1c614b40b6391f1ee45ed14c695e96452b4f39e4dbd9"
Nov 03 20:55:39.607148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:39.607056    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:39.892248 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:39.892157    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:40.127962 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:40.127925    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:40.414450 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:40.414410    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:41.918298 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:41.918246    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:42.053574 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:42.053539    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"]
Nov 03 20:55:42.099681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:42.099645    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496d699c-gvcp4"]
Nov 03 20:55:42.116373 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:42.116338    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86db3616-4842-433e-adc9-61d51fe488da" path="/var/lib/kubelet/pods/86db3616-4842-433e-adc9-61d51fe488da/volumes"
Nov 03 20:55:42.126577 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:42.126545    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:43.174960 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:43.174912    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:43.376897 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:43.376855    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:44.027899 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:44.027856    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:44.278554 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:44.278441    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"]
Nov 03 20:55:44.674060 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:44.674025    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:44.823027 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:44.822988    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:45.548231 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:45.548116    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:45.816846 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:45.816750    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:45.989635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:45.989568    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:46.057119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:46.056148    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"]
Nov 03 20:55:46.124902 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:46.124862    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-43693d42-7vqg5"]
Nov 03 20:55:46.158433 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:46.158400    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:46.256420 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:46.256387    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56b34a8-2a7e-406f-9596-56bf6651b982" path="/var/lib/kubelet/pods/e56b34a8-2a7e-406f-9596-56bf6651b982/volumes"
Nov 03 20:55:46.781675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:46.781635    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:47.490785 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:47.490741    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:47.911351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:47.911311    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:48.486679 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:55:48.486644    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:55:49.278962 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:49.278921    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2" podUID="421e65d1-3d4a-4578-bcd2-3d942f7f0082" containerName="payload-processing" containerID="cri-o://18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329" gracePeriod=25
Nov 03 20:55:49.569537 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:49.569507    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:55:49.646751 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:49.646712    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421e65d1-3d4a-4578-bcd2-3d942f7f0082-kube-api-access-qqfmw\" (UniqueName: \"kubernetes.io/projected/421e65d1-3d4a-4578-bcd2-3d942f7f0082-kube-api-access-qqfmw\") pod \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\" (UID: \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\") "
Nov 03 20:55:49.646931 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:49.646859    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421e65d1-3d4a-4578-bcd2-3d942f7f0082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421e65d1-3d4a-4578-bcd2-3d942f7f0082-plugins-config-volume\") pod \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\" (UID: \"421e65d1-3d4a-4578-bcd2-3d942f7f0082\") "
Nov 03 20:55:49.647238 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:49.647214    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421e65d1-3d4a-4578-bcd2-3d942f7f0082-plugins-config-volume" pod "421e65d1-3d4a-4578-bcd2-3d942f7f0082" (UID: "421e65d1-3d4a-4578-bcd2-3d942f7f0082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:49.647655 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:49.647639    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421e65d1-3d4a-4578-bcd2-3d942f7f0082-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:49.649181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:49.649160    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421e65d1-3d4a-4578-bcd2-3d942f7f0082-kube-api-access-qqfmw" pod "421e65d1-3d4a-4578-bcd2-3d942f7f0082" (UID: "421e65d1-3d4a-4578-bcd2-3d942f7f0082"). InnerVolumeSpecName "kube-api-access-qqfmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:49.749487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:49.749447    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqfmw\" (UniqueName: \"kubernetes.io/projected/421e65d1-3d4a-4578-bcd2-3d942f7f0082-kube-api-access-qqfmw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:50.217268 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.217233    5196 generic.go:356] "Generic (PLEG): container finished" podID="421e65d1-3d4a-4578-bcd2-3d942f7f0082" containerID="18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329" exitCode=0
Nov 03 20:55:50.217451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.217294    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"
Nov 03 20:55:50.217451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.217315    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2" event={"ID":"421e65d1-3d4a-4578-bcd2-3d942f7f0082","Type":"ContainerDied","Data":"18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329"}
Nov 03 20:55:50.217451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.217378    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2" event={"ID":"421e65d1-3d4a-4578-bcd2-3d942f7f0082","Type":"ContainerDied","Data":"2c9281d24eec87d31b5a19370e23e66d2b238272f40378eaab3656bc24bce09e"}
Nov 03 20:55:50.217451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.217404    5196 scope.go:122] "RemoveContainer" containerID="18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329"
Nov 03 20:55:50.230278 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.230162    5196 scope.go:122] "RemoveContainer" containerID="18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329"
Nov 03 20:55:50.230575 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:55:50.230551    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329\": container with ID starting with 18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329 not found: ID does not exist" containerID="18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329"
Nov 03 20:55:50.230657 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.230584    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329"} err="failed to get container status \"18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329\": rpc error: code = NotFound desc = could not find container \"18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329\": container with ID starting with 18147392096f10cf08a1ebc8ad4c6205336cd4b404050ade685f5f011ff1d329 not found: ID does not exist"
Nov 03 20:55:50.378218 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.378133    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"]
Nov 03 20:55:50.423946 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:50.423913    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3e5cd62-774b7d9457-hvqd2"]
Nov 03 20:55:51.035895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:51.035859    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:51.205643 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:51.205607    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:51.370213 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:51.370165    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:52.112483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:52.112442    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421e65d1-3d4a-4578-bcd2-3d942f7f0082" path="/var/lib/kubelet/pods/421e65d1-3d4a-4578-bcd2-3d942f7f0082/volumes"
Nov 03 20:55:54.282635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:54.282604    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:54.321597 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:54.321566    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:55:54.467941 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:54.467787    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:56.822116 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:56.816760    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:58.046772 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:58.046734    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"]
Nov 03 20:55:58.093840 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:58.093804    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58eda9f-qz6zg"]
Nov 03 20:55:58.112200 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:55:58.112166    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c684363-f63c-41d5-af34-d3232e1feef0" path="/var/lib/kubelet/pods/6c684363-f63c-41d5-af34-d3232e1feef0/volumes"
Nov 03 20:56:01.096406 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:01.096347    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:02.728583 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:02.728535    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:02.982586 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:02.982493    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:03.133302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:03.133265    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:03.213034 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:03.212992    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"]
Nov 03 20:56:03.259854 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:03.259769    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1a55d7db-gcm89"]
Nov 03 20:56:03.350981 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:03.350940    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:04.111821 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:04.111790    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6504485-a679-49ad-8633-74677c3b5205" path="/var/lib/kubelet/pods/b6504485-a679-49ad-8633-74677c3b5205/volumes"
Nov 03 20:56:04.322412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:04.322380    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:04.483189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:04.483074    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:04.641198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:04.641162    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:07.525665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:07.525623    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:09.058780 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:09.058744    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:09.285421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:09.285391    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:09.466059 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:09.466027    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:09.595878 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:09.595842    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"]
Nov 03 20:56:09.643396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:09.643364    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3e5cd62-xbvc2"]
Nov 03 20:56:09.970380 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:09.970345    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:10.112033 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:10.112000    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5cb8b8c-6a6c-4c23-8761-002d3a831053" path="/var/lib/kubelet/pods/b5cb8b8c-6a6c-4c23-8761-002d3a831053/volumes"
Nov 03 20:56:14.297849 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:14.297805    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:15.886376 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:15.886340    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:16.054615 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:16.054572    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:16.207765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:16.207673    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:16.671802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:16.671764    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:16.863078 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:16.863046    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:17.142472 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:17.142425    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:18.046254 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:18.046216    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"]
Nov 03 20:56:18.093420 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:18.093388    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e0fa4-6xgwf"]
Nov 03 20:56:18.111881 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:18.111849    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fcf07a-e119-43a5-9fdd-1d922e1c476e" path="/var/lib/kubelet/pods/c5fcf07a-e119-43a5-9fdd-1d922e1c476e/volumes"
Nov 03 20:56:18.488352 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:56:18.488317    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:56:19.047496 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:19.047456    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:21.047637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:21.047598    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"]
Nov 03 20:56:21.094599 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:21.094561    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42c1d91e-k4bt5"]
Nov 03 20:56:22.111883 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:22.111849    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7" path="/var/lib/kubelet/pods/492b3ec0-c13e-49ba-bc57-96e4a8dd5ca7/volumes"
Nov 03 20:56:22.885940 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:22.885898    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:23.042270 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:23.042237    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:23.338199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:23.338161    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:23.699873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:23.699740    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:23.905087 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:23.905018    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:24.057522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.056904    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:24.207802 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.207758    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:24.592797 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.592760    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"]
Nov 03 20:56:24.593972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.593947    5196 cpu_manager.go:414] "containerMap: removing container" podUID="421e65d1-3d4a-4578-bcd2-3d942f7f0082" containerName="payload-processing"
Nov 03 20:56:24.594072 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.593976    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421e65d1-3d4a-4578-bcd2-3d942f7f0082" containerName="payload-processing"
Nov 03 20:56:24.594072 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.594006    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b5cb8b8c-6a6c-4c23-8761-002d3a831053" containerName="revoke-keys"
Nov 03 20:56:24.594072 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.594014    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cb8b8c-6a6c-4c23-8761-002d3a831053" containerName="revoke-keys"
Nov 03 20:56:24.594256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.594242    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="421e65d1-3d4a-4578-bcd2-3d942f7f0082" containerName="payload-processing"
Nov 03 20:56:24.594295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.594262    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cb8b8c-6a6c-4c23-8761-002d3a831053" containerName="revoke-keys"
Nov 03 20:56:24.601543 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.601514    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659041    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659150    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659203    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659442 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659329    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659514 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659480    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659581 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659545    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659639 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659580    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659732 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659711    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhv77\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-kube-api-access-rhv77\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.659977 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.659940    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.669904 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.666880    5196 scope.go:122] "RemoveContainer" containerID="cb97502cf8813a11eafe2312af3f77e63f171b350196c0c610432fda83402ed3"
Nov 03 20:56:24.691566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.691535    5196 scope.go:122] "RemoveContainer" containerID="d3df53ecd04946e7245a24771161705a90fa544e3f08760fa48777d5715a58b5"
Nov 03 20:56:24.709687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.709651    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"]
Nov 03 20:56:24.763804 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.763770    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.763985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.763878    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.763985 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.763944    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764050    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764186 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764149    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764318 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764270    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhv77\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-kube-api-access-rhv77\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764318 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764292    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764438 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764407    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764431    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764508    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764665 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764513    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764730 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764691    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764730 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764700    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.764843 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.764764    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.766763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.766731    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.766990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.766973    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.817547 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.817506    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.817726 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.817583    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhv77\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-kube-api-access-rhv77\") pod \"maas-default-gateway-openshift-default-696f47c6c-nzzws\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:24.925829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:24.925747    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:25.366368 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:25.366341    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"]
Nov 03 20:56:25.367885 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:56:25.367858    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7a8f53_7ecf_4333_b747_d8662ea2bdeb.slice/crio-cbd22fa46a5ba7e8785cbc6fc086e4e29812bc015917f6859ab2a645c8ce72c6 WatchSource:0}: Error finding container cbd22fa46a5ba7e8785cbc6fc086e4e29812bc015917f6859ab2a645c8ce72c6: Status 404 returned error can't find the container with id cbd22fa46a5ba7e8785cbc6fc086e4e29812bc015917f6859ab2a645c8ce72c6
Nov 03 20:56:25.370926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:25.370776    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:56:25.370926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:25.370865    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:56:25.370926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:25.370909    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:56:25.724917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:25.724816    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" event={"ID":"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb","Type":"ContainerStarted","Data":"85163c644e145cf696074fcf96b6b654b0f236001799225c35473e389e52a7f7"}
Nov 03 20:56:25.724917 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:25.724889    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" event={"ID":"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb","Type":"ContainerStarted","Data":"cbd22fa46a5ba7e8785cbc6fc086e4e29812bc015917f6859ab2a645c8ce72c6"}
Nov 03 20:56:25.840614 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:25.840562    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podStartSLOduration=1.840545544 podStartE2EDuration="1.840545544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:25.835598656 +0000 UTC m=+1625.839557384" watchObservedRunningTime="2026-11-03 20:56:25.840545544 +0000 UTC m=+1625.844504265"
Nov 03 20:56:25.928936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:25.928867    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:26.432062 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:26.432024    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:26.928239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:26.928200    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-nzzws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:56:26.928683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:26.928257    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 20:56:27.926490 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:27.926460    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-nzzws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:56:27.926675 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:27.926517    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 20:56:28.660029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:28.659982    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:28.926666 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:28.926582    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-nzzws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:56:28.926666 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:28.926649    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 20:56:29.926701 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:29.926665    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-nzzws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:56:29.927265 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:29.926741    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 20:56:30.926866 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:30.926825    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-nzzws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:56:30.927362 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:30.926884    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 20:56:31.926978 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:31.926935    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-nzzws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:56:31.927431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:31.926996    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 20:56:32.067588 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:32.067546    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:32.926825 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:32.926785    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-nzzws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:56:32.927004 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:32.926865    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 03 20:56:33.490436 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:33.490401    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:34.457050 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:34.457021    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:34.457634 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:34.457615    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:34.458556 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:34.458536    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 20:56:38.609172 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:38.609046    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:38.765139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:38.765080    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:39.047457 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:39.047356    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"]
Nov 03 20:56:39.087140 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:39.087086    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d4a7fa-2gbf9"]
Nov 03 20:56:39.762685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:39.762650    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:40.050086 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:40.049998    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"]
Nov 03 20:56:40.096413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:40.096380    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"]
Nov 03 20:56:40.112942 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:40.112907    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef96d9d8-7a54-4375-93ae-46fc0916677a" path="/var/lib/kubelet/pods/ef96d9d8-7a54-4375-93ae-46fc0916677a/volumes"
Nov 03 20:56:40.113350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:40.113324    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"]
Nov 03 20:56:40.113396 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:40.113366    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"]
Nov 03 20:56:40.155586 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:56:40.155546    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-f40841e4-r74vw\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="49d18462-f811-4f8a-8c1f-3b894975231b" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40841e4-r74vw"
Nov 03 20:56:40.203785 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:56:40.203747    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-046108f4-dqq7s\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="2ba6a943-fec3-4cee-8ec5-8687bb2308bb" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046108f4-dqq7s"
Nov 03 20:56:40.507720 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:40.507674    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:40.669019 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:40.668982    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:40.819683 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:40.819655    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:41.050685 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:41.050654    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"]
Nov 03 20:56:41.097828 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:41.097741    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81402c5e-khb4w"]
Nov 03 20:56:42.126995 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:42.126959    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba6a943-fec3-4cee-8ec5-8687bb2308bb" path="/var/lib/kubelet/pods/2ba6a943-fec3-4cee-8ec5-8687bb2308bb/volumes"
Nov 03 20:56:42.127733 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:42.127710    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d18462-f811-4f8a-8c1f-3b894975231b" path="/var/lib/kubelet/pods/49d18462-f811-4f8a-8c1f-3b894975231b/volumes"
Nov 03 20:56:42.128558 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:42.128538    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909" path="/var/lib/kubelet/pods/b0b7ca98-f1d6-42eb-97b3-ddc4c6fbe909/volumes"
Nov 03 20:56:43.737674 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:43.737636    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:43.920739 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:43.920702    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:44.101366 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.101320    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:44.138977 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.138935    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"]
Nov 03 20:56:44.146081 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.146050    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:44.200769 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.200732    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"]
Nov 03 20:56:44.242123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.241012    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-45f792-dockercfg-q6bdr\""
Nov 03 20:56:44.242123 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.241150    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-45f792\""
Nov 03 20:56:44.306115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.305016    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxfb\" (UniqueName: \"kubernetes.io/projected/082ac835-40a4-4990-865f-91397c4bad21-kube-api-access-8vxfb\") pod \"payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2\" (UID: \"082ac835-40a4-4990-865f-91397c4bad21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:44.306115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.305239    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082ac835-40a4-4990-865f-91397c4bad21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2\" (UID: \"082ac835-40a4-4990-865f-91397c4bad21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:44.426752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.408350    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082ac835-40a4-4990-865f-91397c4bad21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2\" (UID: \"082ac835-40a4-4990-865f-91397c4bad21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:44.426752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.408672    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxfb\" (UniqueName: \"kubernetes.io/projected/082ac835-40a4-4990-865f-91397c4bad21-kube-api-access-8vxfb\") pod \"payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2\" (UID: \"082ac835-40a4-4990-865f-91397c4bad21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:44.426752 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.410472    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082ac835-40a4-4990-865f-91397c4bad21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2\" (UID: \"082ac835-40a4-4990-865f-91397c4bad21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:44.449165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.446566    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"]
Nov 03 20:56:44.458350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.456730    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:44.458350 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.457839    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"]
Nov 03 20:56:44.483888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.468027    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxfb\" (UniqueName: \"kubernetes.io/projected/082ac835-40a4-4990-865f-91397c4bad21-kube-api-access-8vxfb\") pod \"payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2\" (UID: \"082ac835-40a4-4990-865f-91397c4bad21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:44.483888 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.468330    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:44.518150 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.513475    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg8q5\" (UniqueName: \"kubernetes.io/projected/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-kube-api-access-vg8q5\") pod \"payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6\" (UID: \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:44.518150 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.513691    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6\" (UID: \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:44.617583 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.616234    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg8q5\" (UniqueName: \"kubernetes.io/projected/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-kube-api-access-vg8q5\") pod \"payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6\" (UID: \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:44.617583 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.616483    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6\" (UID: \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:44.619038 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.618814    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6\" (UID: \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:44.674081 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.674053    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg8q5\" (UniqueName: \"kubernetes.io/projected/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-kube-api-access-vg8q5\") pod \"payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6\" (UID: \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:44.848035 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.847999    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:44.890585 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:44.890555    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"]
Nov 03 20:56:44.893075 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:56:44.893034    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082ac835_40a4_4990_865f_91397c4bad21.slice/crio-7d417864f7e1c1a0ba3a9e4bd4cdf445926cc86632d0dc0fb4e5fdb236561742 WatchSource:0}: Error finding container 7d417864f7e1c1a0ba3a9e4bd4cdf445926cc86632d0dc0fb4e5fdb236561742: Status 404 returned error can't find the container with id 7d417864f7e1c1a0ba3a9e4bd4cdf445926cc86632d0dc0fb4e5fdb236561742
Nov 03 20:56:45.031139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:45.030044    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2" event={"ID":"082ac835-40a4-4990-865f-91397c4bad21","Type":"ContainerStarted","Data":"4d370ed8f4c3f318597520fe3c6b87ade206aeaf2f3fbc5be44331381338bc59"}
Nov 03 20:56:45.031139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:45.030169    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2" event={"ID":"082ac835-40a4-4990-865f-91397c4bad21","Type":"ContainerStarted","Data":"7d417864f7e1c1a0ba3a9e4bd4cdf445926cc86632d0dc0fb4e5fdb236561742"}
Nov 03 20:56:45.031139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:45.030328    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:45.078990 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:45.078955    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:45.140250 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:45.140130    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2" podStartSLOduration=1.140103123 podStartE2EDuration="1.140103123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:45.137721904 +0000 UTC m=+1645.141680652" watchObservedRunningTime="2026-11-03 20:56:45.140103123 +0000 UTC m=+1645.144061841"
Nov 03 20:56:45.238681 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:45.238651    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"]
Nov 03 20:56:45.241285 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:56:45.241257    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7efd8bf_e3ab_4041_91cf_5719ee35c4b6.slice/crio-48d4e8e7dc5833e656f06279ff615ad610154f4a43e4ffa57bccd4f1d7e60423 WatchSource:0}: Error finding container 48d4e8e7dc5833e656f06279ff615ad610154f4a43e4ffa57bccd4f1d7e60423: Status 404 returned error can't find the container with id 48d4e8e7dc5833e656f06279ff615ad610154f4a43e4ffa57bccd4f1d7e60423
Nov 03 20:56:45.287377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:45.286867    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:46.023445 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.023402    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:46.043171 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.043131    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6" event={"ID":"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6","Type":"ContainerStarted","Data":"3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f"}
Nov 03 20:56:46.043385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.043200    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6" event={"ID":"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6","Type":"ContainerStarted","Data":"48d4e8e7dc5833e656f06279ff615ad610154f4a43e4ffa57bccd4f1d7e60423"}
Nov 03 20:56:46.043385 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.043232    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:46.046599 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.046572    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"]
Nov 03 20:56:46.092836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.092799    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a25245-6kt2x"]
Nov 03 20:56:46.121578 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.121548    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730becb1-edf1-4805-82e2-093baaa9446d" path="/var/lib/kubelet/pods/730becb1-edf1-4805-82e2-093baaa9446d/volumes"
Nov 03 20:56:46.151828 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.151761    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6" podStartSLOduration=2.151746142 podStartE2EDuration="2.151746142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:46.149842752 +0000 UTC m=+1646.153801509" watchObservedRunningTime="2026-11-03 20:56:46.151746142 +0000 UTC m=+1646.155704910"
Nov 03 20:56:46.176745 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:46.176709    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:48.488535 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:56:48.488496    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:56:49.201915 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:49.201874    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:53.376799 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:53.376760    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:53.534008 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:53.533975    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:53.812555 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:53.812460    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:54.047082 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:54.047034    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"]
Nov 03 20:56:54.093679 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:54.093639    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8784fa-vf44f"]
Nov 03 20:56:54.112866 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:54.112834    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ad5ec4-ea32-4961-90df-82b5b6ddb920" path="/var/lib/kubelet/pods/a6ad5ec4-ea32-4961-90df-82b5b6ddb920/volumes"
Nov 03 20:56:54.420303 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:54.420201    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:55.046782 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:55.046742    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"]
Nov 03 20:56:55.092528 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:55.092495    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f054dfe-tq4zp"]
Nov 03 20:56:56.046763 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:56.046729    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"]
Nov 03 20:56:56.047216 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:56.047022    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 20:56:56.092969 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:56.092934    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a67aed-2d7sm"]
Nov 03 20:56:56.112371 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:56.112334    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462cb87f-6a9b-47fa-87c5-8f783a7650fb" path="/var/lib/kubelet/pods/462cb87f-6a9b-47fa-87c5-8f783a7650fb/volumes"
Nov 03 20:56:56.112971 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:56.112946    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e280ce-b242-416e-bf3d-e13cd16fa209" path="/var/lib/kubelet/pods/e1e280ce-b242-416e-bf3d-e13cd16fa209/volumes"
Nov 03 20:56:57.057521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:57.057491    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 20:56:59.461288 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:59.461250    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:59.662582 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:59.662536    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:59.824786 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:56:59.824749    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:00.045829 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:00.045796    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"]
Nov 03 20:57:00.092500 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:00.092416    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8fbe-qjb8t"]
Nov 03 20:57:00.113405 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:00.113363    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc601ea-c169-4823-93a1-0815589780dd" path="/var/lib/kubelet/pods/1dc601ea-c169-4823-93a1-0815589780dd/volumes"
Nov 03 20:57:04.174909 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:04.174874    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:06.046858 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:06.046820    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"]
Nov 03 20:57:06.094351 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:06.094309    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b3d7910-5lns9"]
Nov 03 20:57:06.116323 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:06.116292    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7818f4-eb56-4992-abde-a0fa52f81869" path="/var/lib/kubelet/pods/6f7818f4-eb56-4992-abde-a0fa52f81869/volumes"
Nov 03 20:57:08.953755 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:08.953715    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:09.497046 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:09.497006    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:57:13.760788 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:13.760748    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:15.045968 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:15.045933    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"]
Nov 03 20:57:15.092687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:15.092647    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8179659-9fp2z"]
Nov 03 20:57:16.051312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:16.051281    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"]
Nov 03 20:57:16.097842 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:16.097807    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-978212fb-l5r4k"]
Nov 03 20:57:16.111549 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:16.111513    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a76380-07a5-487e-a82e-23dd52b7cfd5" path="/var/lib/kubelet/pods/86a76380-07a5-487e-a82e-23dd52b7cfd5/volumes"
Nov 03 20:57:16.112047 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:16.112032    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22f1f1a-d59b-476d-8474-d6d611be0d64" path="/var/lib/kubelet/pods/a22f1f1a-d59b-476d-8474-d6d611be0d64/volumes"
Nov 03 20:57:17.049158 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:17.049116    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"]
Nov 03 20:57:17.096115 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:17.096063    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"]
Nov 03 20:57:17.101756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:17.101721    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9daf6c-gc74h"]
Nov 03 20:57:17.107622 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:17.107593    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4902ea-gtcvv"]
Nov 03 20:57:18.114545 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:18.114501    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7082949e-6d4d-4b6f-8e31-eae5d192f2f4" path="/var/lib/kubelet/pods/7082949e-6d4d-4b6f-8e31-eae5d192f2f4/volumes"
Nov 03 20:57:18.115294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:18.115270    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c79042c-40e8-463b-8033-73ccfb31f01a" path="/var/lib/kubelet/pods/7c79042c-40e8-463b-8033-73ccfb31f01a/volumes"
Nov 03 20:57:18.487295 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:57:18.487203    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:57:18.693145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:18.693087    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:23.167910 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.167871    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"]
Nov 03 20:57:23.174469 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.174447    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.223482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.223446    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"]
Nov 03 20:57:23.264032 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.264000    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:57:23.264309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.264284    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:57:23.341280 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.341243    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce737530-af1c-4219-8495-543273b84d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.341466 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.341289    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce737530-af1c-4219-8495-543273b84d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.341593 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.341570    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp4jw\" (UniqueName: \"kubernetes.io/projected/ce737530-af1c-4219-8495-543273b84d3e-kube-api-access-jp4jw\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.443084 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.442981    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce737530-af1c-4219-8495-543273b84d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.443317 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.443302    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp4jw\" (UniqueName: \"kubernetes.io/projected/ce737530-af1c-4219-8495-543273b84d3e-kube-api-access-jp4jw\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.443470 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.443457    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce737530-af1c-4219-8495-543273b84d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.443878 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.443852    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce737530-af1c-4219-8495-543273b84d3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.446160 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.446137    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce737530-af1c-4219-8495-543273b84d3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.497507 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.497471    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp4jw\" (UniqueName: \"kubernetes.io/projected/ce737530-af1c-4219-8495-543273b84d3e-kube-api-access-jp4jw\") pod \"maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:23.769656 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.769572    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:23.814138 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:23.813460    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:24.236792 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:24.236759    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"]
Nov 03 20:57:24.240639 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:57:24.240612    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce737530_af1c_4219_8495_543273b84d3e.slice/crio-8c7729770c8655eed9c9bf9c24349db9590c0912ba3149dec8ac78579ddcde19 WatchSource:0}: Error finding container 8c7729770c8655eed9c9bf9c24349db9590c0912ba3149dec8ac78579ddcde19: Status 404 returned error can't find the container with id 8c7729770c8655eed9c9bf9c24349db9590c0912ba3149dec8ac78579ddcde19
Nov 03 20:57:24.523762 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:24.523673    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7" event={"ID":"ce737530-af1c-4219-8495-543273b84d3e","Type":"ContainerStarted","Data":"8c7729770c8655eed9c9bf9c24349db9590c0912ba3149dec8ac78579ddcde19"}
Nov 03 20:57:24.757471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:24.757435    5196 scope.go:122] "RemoveContainer" containerID="4dcb347691ccdfb83673ac198b67211bc7438f4d376812d2fb995f2598ee4d21"
Nov 03 20:57:24.769972 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:24.769941    5196 scope.go:122] "RemoveContainer" containerID="dddd4922e8997b3ecb9bc909869c979735f14f65a02fdffa7a6e25d7a534d94d"
Nov 03 20:57:24.781987 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:24.781965    5196 scope.go:122] "RemoveContainer" containerID="5cea8e4b394908f00e93fc11355269cfe9f5131541cefc87a9c3b196b9184a71"
Nov 03 20:57:24.794272 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:24.794248    5196 scope.go:122] "RemoveContainer" containerID="7432c2eb6baf548f243d754b00e261671e4986b3a093b2abfaa5063d69f17c0c"
Nov 03 20:57:25.536775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:25.536692    5196 generic.go:356] "Generic (PLEG): container finished" podID="ce737530-af1c-4219-8495-543273b84d3e" containerID="60bdb26110bb3b40e6135c87f31fa350559bbcf1727736d600633a1d1b9240d7" exitCode=0
Nov 03 20:57:25.536775 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:25.536748    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7" event={"ID":"ce737530-af1c-4219-8495-543273b84d3e","Type":"ContainerDied","Data":"60bdb26110bb3b40e6135c87f31fa350559bbcf1727736d600633a1d1b9240d7"}
Nov 03 20:57:26.054861 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.054824    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"]
Nov 03 20:57:26.060575 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.060540    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fdd0566-q5tkk"]
Nov 03 20:57:26.114992 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.114955    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c210b55-ce4e-4b6f-b069-9b37c074b52f" path="/var/lib/kubelet/pods/1c210b55-ce4e-4b6f-b069-9b37c074b52f/volumes"
Nov 03 20:57:26.739826 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.739799    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:26.798347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.798249    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce737530-af1c-4219-8495-543273b84d3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce737530-af1c-4219-8495-543273b84d3e-maas-api-openshift-ca\") pod \"ce737530-af1c-4219-8495-543273b84d3e\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") "
Nov 03 20:57:26.798347 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.798338    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce737530-af1c-4219-8495-543273b84d3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce737530-af1c-4219-8495-543273b84d3e-maas-api-xks-ca\") pod \"ce737530-af1c-4219-8495-543273b84d3e\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") "
Nov 03 20:57:26.798589 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.798506    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce737530-af1c-4219-8495-543273b84d3e-kube-api-access-jp4jw\" (UniqueName: \"kubernetes.io/projected/ce737530-af1c-4219-8495-543273b84d3e-kube-api-access-jp4jw\") pod \"ce737530-af1c-4219-8495-543273b84d3e\" (UID: \"ce737530-af1c-4219-8495-543273b84d3e\") "
Nov 03 20:57:26.798721 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.798694    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce737530-af1c-4219-8495-543273b84d3e-maas-api-openshift-ca" pod "ce737530-af1c-4219-8495-543273b84d3e" (UID: "ce737530-af1c-4219-8495-543273b84d3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:26.799108 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.799069    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce737530-af1c-4219-8495-543273b84d3e-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:26.800744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.800719    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce737530-af1c-4219-8495-543273b84d3e-kube-api-access-jp4jw" pod "ce737530-af1c-4219-8495-543273b84d3e" (UID: "ce737530-af1c-4219-8495-543273b84d3e"). InnerVolumeSpecName "kube-api-access-jp4jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:26.800744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.800734    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce737530-af1c-4219-8495-543273b84d3e-maas-api-xks-ca" pod "ce737530-af1c-4219-8495-543273b84d3e" (UID: "ce737530-af1c-4219-8495-543273b84d3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:57:26.901044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.900995    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp4jw\" (UniqueName: \"kubernetes.io/projected/ce737530-af1c-4219-8495-543273b84d3e-kube-api-access-jp4jw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:26.901044 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:26.901033    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce737530-af1c-4219-8495-543273b84d3e-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:27.053024 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:27.052939    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"]
Nov 03 20:57:27.060458 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:27.060425    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78a14bc-xw9hq"]
Nov 03 20:57:27.556578 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:27.556543    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"
Nov 03 20:57:27.556768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:27.556581    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7" event={"ID":"ce737530-af1c-4219-8495-543273b84d3e","Type":"ContainerDied","Data":"8c7729770c8655eed9c9bf9c24349db9590c0912ba3149dec8ac78579ddcde19"}
Nov 03 20:57:27.556768 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:27.556632    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c7729770c8655eed9c9bf9c24349db9590c0912ba3149dec8ac78579ddcde19"
Nov 03 20:57:28.112431 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:28.112388    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b35dcf3-37da-4854-aee6-abdb8fb71f30" path="/var/lib/kubelet/pods/9b35dcf3-37da-4854-aee6-abdb8fb71f30/volumes"
Nov 03 20:57:28.803227 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:28.803187    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:28.970412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:28.970374    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:29.179749 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:29.179704    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:34.101388 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:34.101340    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:36.205012 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:36.204977    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:36.375243 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:36.365421    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:39.047391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:39.047356    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"]
Nov 03 20:57:39.094107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:39.094064    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96692c87-f6bt6"]
Nov 03 20:57:39.197312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:39.197267    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:40.047757 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:40.047713    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"]
Nov 03 20:57:40.094862 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:40.094823    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1658d065-tsr88"]
Nov 03 20:57:40.113285 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:40.113245    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe9721b-7852-454a-869a-94aabb82a390" path="/var/lib/kubelet/pods/2fe9721b-7852-454a-869a-94aabb82a390/volumes"
Nov 03 20:57:40.113871 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:40.113854    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f48dad2-b50a-4b23-be53-6c2e082cc2fb" path="/var/lib/kubelet/pods/4f48dad2-b50a-4b23-be53-6c2e082cc2fb/volumes"
Nov 03 20:57:43.045662 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:43.045625    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"]
Nov 03 20:57:43.092525 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:43.092477    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d42fd3-9r74q"]
Nov 03 20:57:44.112793 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.112748    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e198cf11-c062-47a5-a83f-b109821d5b58" path="/var/lib/kubelet/pods/e198cf11-c062-47a5-a83f-b109821d5b58/volumes"
Nov 03 20:57:44.285323 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.285287    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"]
Nov 03 20:57:44.286164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.286143    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ce737530-af1c-4219-8495-543273b84d3e" containerName="revoke-keys"
Nov 03 20:57:44.286164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.286160    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce737530-af1c-4219-8495-543273b84d3e" containerName="revoke-keys"
Nov 03 20:57:44.286343 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.286333    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce737530-af1c-4219-8495-543273b84d3e" containerName="revoke-keys"
Nov 03 20:57:44.291491 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.291470    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.342602 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.342565    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"]
Nov 03 20:57:44.354596 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.354564    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:57:44.382853 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.382781    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:57:44.389631 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.389590    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:44.469397 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.469362    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.469624 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.469603    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.469836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.469812    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrqd\" (UniqueName: \"kubernetes.io/projected/7c715dde-6801-4095-af6f-faccefa7464a-kube-api-access-rjrqd\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.571874 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.571834    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.572075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.572041    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrqd\" (UniqueName: \"kubernetes.io/projected/7c715dde-6801-4095-af6f-faccefa7464a-kube-api-access-rjrqd\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.572291 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.572271    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.572983 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.572959    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.574714 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.574694    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.626168 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.626130    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrqd\" (UniqueName: \"kubernetes.io/projected/7c715dde-6801-4095-af6f-faccefa7464a-kube-api-access-rjrqd\") pod \"maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:44.909145 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:44.909111    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:45.296774 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:45.296744    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"]
Nov 03 20:57:45.300160 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:57:45.300126    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c715dde_6801_4095_af6f_faccefa7464a.slice/crio-3c085099bbe1f30878d67b96cc340be65a868672cd2a887024c0e6b683fb0892 WatchSource:0}: Error finding container 3c085099bbe1f30878d67b96cc340be65a868672cd2a887024c0e6b683fb0892: Status 404 returned error can't find the container with id 3c085099bbe1f30878d67b96cc340be65a868672cd2a887024c0e6b683fb0892
Nov 03 20:57:45.401908 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:45.401861    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:45.788530 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:45.788491    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2" event={"ID":"7c715dde-6801-4095-af6f-faccefa7464a","Type":"ContainerStarted","Data":"3c085099bbe1f30878d67b96cc340be65a868672cd2a887024c0e6b683fb0892"}
Nov 03 20:57:46.801820 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:46.801735    5196 generic.go:356] "Generic (PLEG): container finished" podID="7c715dde-6801-4095-af6f-faccefa7464a" containerID="6f879014f57fa9e157d700bbb384a13abe04aa8cf62b64a50b8e4d6493dd4843" exitCode=0
Nov 03 20:57:46.802239 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:46.801858    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2" event={"ID":"7c715dde-6801-4095-af6f-faccefa7464a","Type":"ContainerDied","Data":"6f879014f57fa9e157d700bbb384a13abe04aa8cf62b64a50b8e4d6493dd4843"}
Nov 03 20:57:48.015893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.015861    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:48.127895 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.127859    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c715dde-6801-4095-af6f-faccefa7464a-kube-api-access-rjrqd\" (UniqueName: \"kubernetes.io/projected/7c715dde-6801-4095-af6f-faccefa7464a-kube-api-access-rjrqd\") pod \"7c715dde-6801-4095-af6f-faccefa7464a\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") "
Nov 03 20:57:48.128119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.127990    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-xks-ca\") pod \"7c715dde-6801-4095-af6f-faccefa7464a\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") "
Nov 03 20:57:48.128119 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.128077    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-openshift-ca\") pod \"7c715dde-6801-4095-af6f-faccefa7464a\" (UID: \"7c715dde-6801-4095-af6f-faccefa7464a\") "
Nov 03 20:57:48.128504 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.128474    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-openshift-ca" pod "7c715dde-6801-4095-af6f-faccefa7464a" (UID: "7c715dde-6801-4095-af6f-faccefa7464a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:48.128867 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.128843    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.130386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.130359    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-xks-ca" pod "7c715dde-6801-4095-af6f-faccefa7464a" (UID: "7c715dde-6801-4095-af6f-faccefa7464a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:57:48.130609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.130587    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c715dde-6801-4095-af6f-faccefa7464a-kube-api-access-rjrqd" pod "7c715dde-6801-4095-af6f-faccefa7464a" (UID: "7c715dde-6801-4095-af6f-faccefa7464a"). InnerVolumeSpecName "kube-api-access-rjrqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:48.230412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.230383    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjrqd\" (UniqueName: \"kubernetes.io/projected/7c715dde-6801-4095-af6f-faccefa7464a-kube-api-access-rjrqd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.230412 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.230409    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c715dde-6801-4095-af6f-faccefa7464a-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.487496 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:57:48.487405    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:57:48.824334 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.824301    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"
Nov 03 20:57:48.824502 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.824321    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2" event={"ID":"7c715dde-6801-4095-af6f-faccefa7464a","Type":"ContainerDied","Data":"3c085099bbe1f30878d67b96cc340be65a868672cd2a887024c0e6b683fb0892"}
Nov 03 20:57:48.824502 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:48.824387    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c085099bbe1f30878d67b96cc340be65a868672cd2a887024c0e6b683fb0892"
Nov 03 20:57:49.595482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:49.595446    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:54.055818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:54.055737    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"]
Nov 03 20:57:54.100984 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:54.100947    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dd62e7-8f6ql"]
Nov 03 20:57:54.112465 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:54.112433    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8150b373-242f-4dde-a30b-293d48b2b3e6" path="/var/lib/kubelet/pods/8150b373-242f-4dde-a30b-293d48b2b3e6/volumes"
Nov 03 20:57:54.369973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:54.369928    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:55.088107 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:55.088046    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"]
Nov 03 20:57:55.093018 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:55.092981    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc12832-sdj8p"]
Nov 03 20:57:56.047670 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:56.047633    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"]
Nov 03 20:57:56.100446 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:56.100406    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf3ec3a-hp644"]
Nov 03 20:57:56.116621 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:56.116583    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295a2c23-a8e9-4929-8716-a99c15ac52b7" path="/var/lib/kubelet/pods/295a2c23-a8e9-4929-8716-a99c15ac52b7/volumes"
Nov 03 20:57:56.117189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:56.117136    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c7b3f5-671f-49c3-a318-b2ee0f23e0a7" path="/var/lib/kubelet/pods/30c7b3f5-671f-49c3-a318-b2ee0f23e0a7/volumes"
Nov 03 20:57:56.117567 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:56.117551    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"]
Nov 03 20:57:56.117609 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:56.117584    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5925af8-g5lqr"]
Nov 03 20:57:56.157515 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:56.157475    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:56.326174 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:56.321684    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:57.240835 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:57.240799    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:58.115673 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:58.115639    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca11177-4162-4f77-8a25-fc2b4fd584f1" path="/var/lib/kubelet/pods/4ca11177-4162-4f77-8a25-fc2b4fd584f1/volumes"
Nov 03 20:57:58.295493 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:58.295455    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:58.898818 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:58.898776    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:59.163363 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:57:59.163265    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:01.662706 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:01.662665    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:02.501697 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:02.501661    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:02.711329 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:02.711287    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:04.225848 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:04.225813    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:05.728778 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:05.728733    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:05.996764 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:05.996679    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:06.127402 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.127367    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"]
Nov 03 20:58:06.128340 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.128319    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7c715dde-6801-4095-af6f-faccefa7464a" containerName="revoke-keys"
Nov 03 20:58:06.128340 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.128338    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c715dde-6801-4095-af6f-faccefa7464a" containerName="revoke-keys"
Nov 03 20:58:06.128522 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.128499    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c715dde-6801-4095-af6f-faccefa7464a" containerName="revoke-keys"
Nov 03 20:58:06.132765 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.132725    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:06.183936 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.183901    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"]
Nov 03 20:58:06.200299 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.199035    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc0d76f-69d0-4539-8989-f58256e9457e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s\" (UID: \"9fc0d76f-69d0-4539-8989-f58256e9457e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:06.200299 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.199210    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htfsd\" (UniqueName: \"kubernetes.io/projected/9fc0d76f-69d0-4539-8989-f58256e9457e-kube-api-access-htfsd\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s\" (UID: \"9fc0d76f-69d0-4539-8989-f58256e9457e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:06.230180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.227364    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-211d68bf\""
Nov 03 20:58:06.230180 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.227571    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-211d68bf-dockercfg-7qnlv\""
Nov 03 20:58:06.323183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.319965    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc0d76f-69d0-4539-8989-f58256e9457e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s\" (UID: \"9fc0d76f-69d0-4539-8989-f58256e9457e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:06.323183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.320064    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htfsd\" (UniqueName: \"kubernetes.io/projected/9fc0d76f-69d0-4539-8989-f58256e9457e-kube-api-access-htfsd\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s\" (UID: \"9fc0d76f-69d0-4539-8989-f58256e9457e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:06.323183 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.321085    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc0d76f-69d0-4539-8989-f58256e9457e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s\" (UID: \"9fc0d76f-69d0-4539-8989-f58256e9457e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:06.343728 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.331239    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"]
Nov 03 20:58:06.357224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.357170    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"]
Nov 03 20:58:06.367181 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.357645    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:06.387750 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.387718    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htfsd\" (UniqueName: \"kubernetes.io/projected/9fc0d76f-69d0-4539-8989-f58256e9457e-kube-api-access-htfsd\") pod \"payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s\" (UID: \"9fc0d76f-69d0-4539-8989-f58256e9457e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:06.421926 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.421851    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-plugins-config-volume\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n\" (UID: \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:06.422228 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.422153    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q8gd\" (UniqueName: \"kubernetes.io/projected/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-kube-api-access-7q8gd\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n\" (UID: \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:06.460451 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.460409    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:06.523956 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.523907    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8gd\" (UniqueName: \"kubernetes.io/projected/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-kube-api-access-7q8gd\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n\" (UID: \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:06.524235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.524207    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-plugins-config-volume\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n\" (UID: \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:06.525904 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.525515    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-plugins-config-volume\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n\" (UID: \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:06.591430 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.582872    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q8gd\" (UniqueName: \"kubernetes.io/projected/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-kube-api-access-7q8gd\") pod \"payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n\" (UID: \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\") " pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:06.715507 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.715470    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:06.886630 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:06.886593    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"]
Nov 03 20:58:07.046751 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:07.046703    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"]
Nov 03 20:58:07.052921 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:07.052886    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cccdecc-9bfhx"]
Nov 03 20:58:07.056292 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:07.056260    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s" event={"ID":"9fc0d76f-69d0-4539-8989-f58256e9457e","Type":"ContainerStarted","Data":"ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2"}
Nov 03 20:58:07.056479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:07.056352    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:07.056479 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:07.056370    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s" event={"ID":"9fc0d76f-69d0-4539-8989-f58256e9457e","Type":"ContainerStarted","Data":"0fa3c3b1784b356ad5da8b89c4d8d9184c61ceb2c7a275b4840e71891b3f2e7b"}
Nov 03 20:58:07.168416 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:07.168387    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"]
Nov 03 20:58:07.171373 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:58:07.171342    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e0e8e5_5d4e_4237_bbe4_a055a747ae58.slice/crio-295b8a44c4e9f6ca9052f627c5c507556e929d8d5c28a013add1e1153b4560a1 WatchSource:0}: Error finding container 295b8a44c4e9f6ca9052f627c5c507556e929d8d5c28a013add1e1153b4560a1: Status 404 returned error can't find the container with id 295b8a44c4e9f6ca9052f627c5c507556e929d8d5c28a013add1e1153b4560a1
Nov 03 20:58:07.173893 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:07.173837    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s" podStartSLOduration=1.1738219380000001 podStartE2EDuration="1.173821938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:07.164886761 +0000 UTC m=+1727.168845503" watchObservedRunningTime="2026-11-03 20:58:07.173821938 +0000 UTC m=+1727.177780663"
Nov 03 20:58:08.046139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.046077    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"]
Nov 03 20:58:08.051633 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.051593    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c9b013-fqzzf"]
Nov 03 20:58:08.068750 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.068715    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n" event={"ID":"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58","Type":"ContainerStarted","Data":"76987d64594c929acdfedf0c154e5c7d9d85ee6f2852ee28890215e472e9a93d"}
Nov 03 20:58:08.068966 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.068775    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n" event={"ID":"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58","Type":"ContainerStarted","Data":"295b8a44c4e9f6ca9052f627c5c507556e929d8d5c28a013add1e1153b4560a1"}
Nov 03 20:58:08.069043 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.068998    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:08.112704 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.112657    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788b1908-6838-497f-9b10-66dd33942d47" path="/var/lib/kubelet/pods/788b1908-6838-497f-9b10-66dd33942d47/volumes"
Nov 03 20:58:08.113391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.113368    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a49a61-1d0e-4b66-90b6-3c4936eb2283" path="/var/lib/kubelet/pods/f0a49a61-1d0e-4b66-90b6-3c4936eb2283/volumes"
Nov 03 20:58:08.175908 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.175845    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n" podStartSLOduration=2.175831379 podStartE2EDuration="2.175831379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:08.174592867 +0000 UTC m=+1728.178551595" watchObservedRunningTime="2026-11-03 20:58:08.175831379 +0000 UTC m=+1728.179790103"
Nov 03 20:58:08.630263 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:08.630211    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:10.157636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:10.157598    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:10.389761 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:10.389730    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:10.534383 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:10.534277    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:10.636645 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:10.636609    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"]
Nov 03 20:58:10.683727 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:10.683692    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e8374c4-krtx7"]
Nov 03 20:58:12.097309 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:12.097259    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:12.112286 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:12.112250    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce737530-af1c-4219-8495-543273b84d3e" path="/var/lib/kubelet/pods/ce737530-af1c-4219-8495-543273b84d3e/volumes"
Nov 03 20:58:12.863877 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:12.863834    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:14.142715 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:14.142670    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:15.014711 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:15.014675    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:15.570291 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:15.570254    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:18.072691 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:18.072661    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 20:58:18.352129 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:18.352016    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:18.490078 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:58:18.490037    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:58:18.516039 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:18.515995    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:18.680075 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:18.679974    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:19.083199 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:19.083170    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 20:58:20.812991 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:20.812960    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:22.670471 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:22.670430    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:23.601878 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:23.601835    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:58:24.859894 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.859861    5196 scope.go:122] "RemoveContainer" containerID="660a39eeb4cba88d0b8448a5031144fc974d64ffcaf1597dbc1213ad75c777f3"
Nov 03 20:58:24.874055 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.874034    5196 scope.go:122] "RemoveContainer" containerID="80be9e584cd4cf807850bd52e461c0374e84a315a893f774b6d54e52eba663dd"
Nov 03 20:58:24.886717 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.886696    5196 scope.go:122] "RemoveContainer" containerID="bb5f5d8d937d82a19533fe901b324692f918344f432d21764072bd5512a2f7e5"
Nov 03 20:58:24.901285 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.901262    5196 scope.go:122] "RemoveContainer" containerID="3f583dcc770a099917b3c38a50e6d2575d92c9d36e95d664736d4d120af3cf93"
Nov 03 20:58:24.917566 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.917542    5196 scope.go:122] "RemoveContainer" containerID="f7add4df3ae9ebb10da8a9703ea03d6d5850a8776991995812252e46c1e30e29"
Nov 03 20:58:24.932890 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.932863    5196 scope.go:122] "RemoveContainer" containerID="c14da6b8a40ad4790cc986e62f2671e5dcc8252ddadf41095854746f994bfc30"
Nov 03 20:58:24.947492 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.947464    5196 scope.go:122] "RemoveContainer" containerID="8c84c47e9c162ccb5b56eca6151aba62f1b2f19f96321249556edae434d369d6"
Nov 03 20:58:24.959508 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.959486    5196 scope.go:122] "RemoveContainer" containerID="4962933513b7cd43d2a22ff8d411c0b79ad1f80d0a5285a5496a7d378010f80d"
Nov 03 20:58:24.971787 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.971750    5196 scope.go:122] "RemoveContainer" containerID="d960c5dacb4b91ed21d633fd244a846af04974fa687ff2d8ad4b834a79d9baeb"
Nov 03 20:58:24.983600 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.983576    5196 scope.go:122] "RemoveContainer" containerID="2572add56b238b6cb18975853dbe3ab07026e2c229d3e407b738209d55aa33e3"
Nov 03 20:58:24.996295 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:24.996271    5196 scope.go:122] "RemoveContainer" containerID="1ffa5b1342bce36491aa3e9dc78469d3ad2c0be933610e28ac794a0b0dec0b03"
Nov 03 20:58:25.011569 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:25.011538    5196 scope.go:122] "RemoveContainer" containerID="2e2b6c7c1b8a2201fe29209c4dcac30adc8e6a4365a083f44f364c4254854f37"
Nov 03 20:58:25.024644 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:25.024619    5196 scope.go:122] "RemoveContainer" containerID="44ec4a474ad6cd86875fddffe6efe3a8adedd9a8396015faf260713a55df6c1b"
Nov 03 20:58:25.038910 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:25.038886    5196 scope.go:122] "RemoveContainer" containerID="d16d7e7087361685bf019f6530ddb5eda68ad4bb4aa60be6677bf7252d15489e"
Nov 03 20:58:25.052148 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:25.052070    5196 scope.go:122] "RemoveContainer" containerID="a45e163ee6b7707f9d75fac9266dd91ed659d9bbb2a5a9c658c2d948c9295138"
Nov 03 20:58:25.065684 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:25.065660    5196 scope.go:122] "RemoveContainer" containerID="f9e60e1e91337bbc4e30437f7ee7f292b1419c9668a9cd67d6f81b98fc50ec3c"
Nov 03 20:58:25.473161 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:25.450658    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:25.606807 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:25.606755    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:27.282314 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:27.281781    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:27.495555 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:27.495516    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:27.713468 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:27.713413    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:27.835952 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:27.835912    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"]
Nov 03 20:58:27.881108 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:27.881058    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc15b1d7-h2cp2"]
Nov 03 20:58:27.911297 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:27.911261    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:28.112029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:28.111996    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c715dde-6801-4095-af6f-faccefa7464a" path="/var/lib/kubelet/pods/7c715dde-6801-4095-af6f-faccefa7464a/volumes"
Nov 03 20:58:28.328029 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:28.327981    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:29.183999 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:29.183963    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:29.348521 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:29.348363    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:29.509698 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:29.509586    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:30.822797 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:30.822746    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:31.063178 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:31.063139    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:32.883747 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:32.883716    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:33.155636 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:33.155542    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:33.302753 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:33.302713    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:35.046033 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:35.045990    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"]
Nov 03 20:58:35.093023 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:35.092988    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f1ad4e-hhfxv"]
Nov 03 20:58:35.649256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:35.649210    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:35.867466 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:35.867430    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:36.112880 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:36.112846    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b411414-ab81-41dc-ac21-0823b35e6200" path="/var/lib/kubelet/pods/3b411414-ab81-41dc-ac21-0823b35e6200/volumes"
Nov 03 20:58:37.563444 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:37.563411    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:37.725290 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:37.725253    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:38.666256 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:38.666216    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:42.411164 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:42.411120    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:42.631643 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:42.631594    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:43.927125 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:43.917589    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:45.072796 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:45.072757    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"]
Nov 03 20:58:45.073473 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:45.073428    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy" containerID="cri-o://2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c" gracePeriod=30
Nov 03 20:58:45.718151 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:45.718073    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:47.970005 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:47.969965    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:48.494333 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:58:48.494287    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:58:48.722348 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:48.722316    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:50.387507 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.387473    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:58:50.491224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491186    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/290be761-fb7e-4715-9a49-856178eb262c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290be761-fb7e-4715-9a49-856178eb262c-istio-podinfo\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.491456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491259    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-data\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.491456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491289    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-credential-socket\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.491456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491326    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-socket\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.491456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491403    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-certs\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.491973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491460    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290be761-fb7e-4715-9a49-856178eb262c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290be761-fb7e-4715-9a49-856178eb262c-istiod-ca-cert\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.491973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491492    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-envoy\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.491973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491516    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-istio-token\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-istio-token\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.491973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491571    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-data" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:50.491973 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491583    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-kube-api-access-l56q9\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-kube-api-access-l56q9\") pod \"290be761-fb7e-4715-9a49-856178eb262c\" (UID: \"290be761-fb7e-4715-9a49-856178eb262c\") "
Nov 03 20:58:50.492329 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.491975    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-socket" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:50.492386 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.492359    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290be761-fb7e-4715-9a49-856178eb262c-istiod-ca-cert" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:50.492517 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.492491    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-certs" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:50.492649 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.492526    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-credential-socket" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:50.493237 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.493202    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.493312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.493247    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.493312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.493264    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.493312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.493279    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.493312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.493293    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290be761-fb7e-4715-9a49-856178eb262c-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.494136 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.494107    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/290be761-fb7e-4715-9a49-856178eb262c-istio-podinfo" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:50.494513 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.494487    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-istio-token" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:50.494778 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.494762    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-envoy" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:50.494862 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.494842    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-kube-api-access-l56q9" pod "290be761-fb7e-4715-9a49-856178eb262c" (UID: "290be761-fb7e-4715-9a49-856178eb262c"). InnerVolumeSpecName "kube-api-access-l56q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:50.596189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.596151    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290be761-fb7e-4715-9a49-856178eb262c-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.596189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.596186    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290be761-fb7e-4715-9a49-856178eb262c-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.596189 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.596195    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.596505 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.596205    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l56q9\" (UniqueName: \"kubernetes.io/projected/290be761-fb7e-4715-9a49-856178eb262c-kube-api-access-l56q9\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:50.963756 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.963723    5196 generic.go:356] "Generic (PLEG): container finished" podID="290be761-fb7e-4715-9a49-856178eb262c" containerID="2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c" exitCode=0
Nov 03 20:58:50.963933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.963794    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"
Nov 03 20:58:50.963933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.963818    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" event={"ID":"290be761-fb7e-4715-9a49-856178eb262c","Type":"ContainerDied","Data":"2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c"}
Nov 03 20:58:50.963933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.963880    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw" event={"ID":"290be761-fb7e-4715-9a49-856178eb262c","Type":"ContainerDied","Data":"513578ea03bec75792ba5fbe29a8bc3f57a17cf49c7e7c994b608a802f9a1c69"}
Nov 03 20:58:50.963933 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.963909    5196 scope.go:122] "RemoveContainer" containerID="2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c"
Nov 03 20:58:50.982618 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.982593    5196 scope.go:122] "RemoveContainer" containerID="2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c"
Nov 03 20:58:50.983002 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:58:50.982977    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c\": container with ID starting with 2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c not found: ID does not exist" containerID="2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c"
Nov 03 20:58:50.983131 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:50.983018    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c"} err="failed to get container status \"2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c\": rpc error: code = NotFound desc = could not find container \"2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c\": container with ID starting with 2f90cf9b94aea6aa5aecd8587b9151532cb62e1af63a60efa201fa4f326eea7c not found: ID does not exist"
Nov 03 20:58:51.126687 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:51.126638    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"]
Nov 03 20:58:51.166497 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:51.166459    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-qtmsw"]
Nov 03 20:58:52.112219 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:52.112182    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290be761-fb7e-4715-9a49-856178eb262c" path="/var/lib/kubelet/pods/290be761-fb7e-4715-9a49-856178eb262c/volumes"
Nov 03 20:58:53.408808 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:53.408769    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:53.572905 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:53.572868    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:53.718224 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:53.718114    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:58.363456 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:58:58.363418    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:03.084487 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:03.084440    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:07.798682 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:07.798642    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:10.983315 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:10.983273    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"]
Nov 03 20:59:10.984294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:10.984272    5196 cpu_manager.go:414] "containerMap: removing container" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy"
Nov 03 20:59:10.984294 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:10.984293    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy"
Nov 03 20:59:10.984464 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:10.984453    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="290be761-fb7e-4715-9a49-856178eb262c" containerName="istio-proxy"
Nov 03 20:59:10.988596 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:10.988575    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.039851 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.039815    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"]
Nov 03 20:59:11.080628 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.080593    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:59:11.080836 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.080675    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 20:59:11.090684 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.090651    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.090839 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.090723    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.090839 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.090770    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxzt\" (UniqueName: \"kubernetes.io/projected/23504e58-91da-43d9-a79a-17b69fe84730-kube-api-access-plxzt\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.192240 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.192189    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.192452 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.192354    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.192452 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.192379    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxzt\" (UniqueName: \"kubernetes.io/projected/23504e58-91da-43d9-a79a-17b69fe84730-kube-api-access-plxzt\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.192970 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.192950    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.195299 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.195273    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.246482 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.246406    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxzt\" (UniqueName: \"kubernetes.io/projected/23504e58-91da-43d9-a79a-17b69fe84730-kube-api-access-plxzt\") pod \"maas-api-revoke-sub-39b5327d-sgx78\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.304040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.304009    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:11.689207 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:11.689049    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"]
Nov 03 20:59:11.692780 ip-10-0-128-49 kubenswrapper[5196]: W1103 20:59:11.692754    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23504e58_91da_43d9_a79a_17b69fe84730.slice/crio-b69750c025983ef903dd5300c6a80b8f85409e817889382fa7832478a26911ca WatchSource:0}: Error finding container b69750c025983ef903dd5300c6a80b8f85409e817889382fa7832478a26911ca: Status 404 returned error can't find the container with id b69750c025983ef903dd5300c6a80b8f85409e817889382fa7832478a26911ca
Nov 03 20:59:12.164689 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:12.164653    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78" event={"ID":"23504e58-91da-43d9-a79a-17b69fe84730","Type":"ContainerStarted","Data":"55b31127592d8ddc3b99bd8047d9fa03ef4e1b5ca20e498a8dd7828f04d6b46c"}
Nov 03 20:59:12.165141 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:12.164726    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78" event={"ID":"23504e58-91da-43d9-a79a-17b69fe84730","Type":"ContainerStarted","Data":"b69750c025983ef903dd5300c6a80b8f85409e817889382fa7832478a26911ca"}
Nov 03 20:59:12.273298 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:12.273223    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78" podStartSLOduration=2.2732069839999998 podStartE2EDuration="2.273206984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:12.270937364 +0000 UTC m=+1792.274896111" watchObservedRunningTime="2026-11-03 20:59:12.273206984 +0000 UTC m=+1792.277165707"
Nov 03 20:59:12.465114 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:12.464238    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:12.652128 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:12.644560    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:12.882223 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:12.882158    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:59:13.008391 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:13.008343    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:13.176139 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:13.176026    5196 generic.go:356] "Generic (PLEG): container finished" podID="23504e58-91da-43d9-a79a-17b69fe84730" containerID="55b31127592d8ddc3b99bd8047d9fa03ef4e1b5ca20e498a8dd7828f04d6b46c" exitCode=0
Nov 03 20:59:13.176593 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:13.176147    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78" event={"ID":"23504e58-91da-43d9-a79a-17b69fe84730","Type":"ContainerDied","Data":"55b31127592d8ddc3b99bd8047d9fa03ef4e1b5ca20e498a8dd7828f04d6b46c"}
Nov 03 20:59:14.425741 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.425712    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:14.604989 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.604938    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-xks-ca\") pod \"23504e58-91da-43d9-a79a-17b69fe84730\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") "
Nov 03 20:59:14.605198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.605185    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23504e58-91da-43d9-a79a-17b69fe84730-kube-api-access-plxzt\" (UniqueName: \"kubernetes.io/projected/23504e58-91da-43d9-a79a-17b69fe84730-kube-api-access-plxzt\") pod \"23504e58-91da-43d9-a79a-17b69fe84730\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") "
Nov 03 20:59:14.605312 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.605284    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-openshift-ca\") pod \"23504e58-91da-43d9-a79a-17b69fe84730\" (UID: \"23504e58-91da-43d9-a79a-17b69fe84730\") "
Nov 03 20:59:14.605692 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.605667    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-openshift-ca" pod "23504e58-91da-43d9-a79a-17b69fe84730" (UID: "23504e58-91da-43d9-a79a-17b69fe84730"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:14.606054 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.605983    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:14.607525 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.607491    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-xks-ca" pod "23504e58-91da-43d9-a79a-17b69fe84730" (UID: "23504e58-91da-43d9-a79a-17b69fe84730"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:59:14.607635 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.607609    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23504e58-91da-43d9-a79a-17b69fe84730-kube-api-access-plxzt" pod "23504e58-91da-43d9-a79a-17b69fe84730" (UID: "23504e58-91da-43d9-a79a-17b69fe84730"). InnerVolumeSpecName "kube-api-access-plxzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:14.707339 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.707297    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23504e58-91da-43d9-a79a-17b69fe84730-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:14.707339 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:14.707336    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plxzt\" (UniqueName: \"kubernetes.io/projected/23504e58-91da-43d9-a79a-17b69fe84730-kube-api-access-plxzt\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:15.197603 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:15.197578    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"
Nov 03 20:59:15.197784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:15.197622    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78" event={"ID":"23504e58-91da-43d9-a79a-17b69fe84730","Type":"ContainerDied","Data":"b69750c025983ef903dd5300c6a80b8f85409e817889382fa7832478a26911ca"}
Nov 03 20:59:15.197784 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:15.197676    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b69750c025983ef903dd5300c6a80b8f85409e817889382fa7832478a26911ca"
Nov 03 20:59:17.421166 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:17.421130    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:18.489460 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:59:18.489418    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:59:20.437153 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:20.437123    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:59:20.444190 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:20.444155    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:59:20.450935 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:20.450912    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 20:59:20.457876 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:20.457847    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 20:59:21.806358 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:21.806319    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:22.148023 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:22.147983    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:25.540235 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.540158    5196 scope.go:122] "RemoveContainer" containerID="d224f314767f6066ee21b5750a6420062382e488ba4aac11a796e27dce3f4ec1"
Nov 03 20:59:25.552259 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.552233    5196 scope.go:122] "RemoveContainer" containerID="b503c2493b6263f12a9dc0ac835d9d96e1a19a2c06d07911c30fcf44b6d32cb1"
Nov 03 20:59:25.563857 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.563835    5196 scope.go:122] "RemoveContainer" containerID="bac6d06991a5b17ef701abee49e051381363332c83facf713f1ea21f16a47985"
Nov 03 20:59:25.575421 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.575399    5196 scope.go:122] "RemoveContainer" containerID="7b948272ad3f4dc1fe42365c1872a49abdfee3757ae0882703c909b75be62bed"
Nov 03 20:59:25.592198 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.592177    5196 scope.go:122] "RemoveContainer" containerID="1d47a9b131b55ff956bd66f7486db403f82ec57a6761d4eeb46cd9a6c8c381cb"
Nov 03 20:59:25.603550 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.603527    5196 scope.go:122] "RemoveContainer" containerID="a7130a0f286e06d80c0d2b20de6aff1703141607952b72a97df4d6e86aa8e308"
Nov 03 20:59:25.615483 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.615456    5196 scope.go:122] "RemoveContainer" containerID="6107d14f5a30d79bd17771b2d22f41be1353d7e94186508316d49c3a47865e60"
Nov 03 20:59:25.628302 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.628280    5196 scope.go:122] "RemoveContainer" containerID="e4128f0d4576b2cc59140b3947754682237a871c0661bf61c74e1fa069482cfe"
Nov 03 20:59:25.639896 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:25.639875    5196 scope.go:122] "RemoveContainer" containerID="23a4957e57abd492f2f6538a5631445907fc4b3a0101610281e8603158f4e5a0"
Nov 03 20:59:26.880595 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:26.875755    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:27.027599 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:27.027562    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:27.197085 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:27.197005    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:27.595040 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:27.595001    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:31.736452 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:31.736413    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:31.908617 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:31.908575    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:32.062000 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:32.061905    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:36.772744 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:36.772701    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:39.831899 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:39.831859    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"]
Nov 03 20:59:39.832377 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:39.832136    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy" containerID="cri-o://cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52" gracePeriod=30
Nov 03 20:59:41.585535 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:41.585502    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:45.130165 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.130082    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:59:45.190334 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190303    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-envoy\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.190524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190374    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-credential-socket\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.190524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190418    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-kube-api-access-6ph7f\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-kube-api-access-6ph7f\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.190524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190480    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-certs\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.190524 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190502    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-data\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.190759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190600    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bdf8d45-c633-456c-a6f0-268812229551-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdf8d45-c633-456c-a6f0-268812229551-istio-podinfo\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.190759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190646    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-socket\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.190759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190724    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-credential-socket" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:45.190759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190738    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-certs" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:45.190759 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190753    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bdf8d45-c633-456c-a6f0-268812229551-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdf8d45-c633-456c-a6f0-268812229551-istiod-ca-cert\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.191036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190793    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-istio-token\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-istio-token\") pod \"3bdf8d45-c633-456c-a6f0-268812229551\" (UID: \"3bdf8d45-c633-456c-a6f0-268812229551\") "
Nov 03 20:59:45.191036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190871    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-data" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:45.191036 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.190976    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-socket" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:45.191236 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.191126    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdf8d45-c633-456c-a6f0-268812229551-istiod-ca-cert" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:45.193076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.192991    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.193076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.193020    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.193076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.193039    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.193076 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.193070    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.193489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.193111    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdf8d45-c633-456c-a6f0-268812229551-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.193489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.193282    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bdf8d45-c633-456c-a6f0-268812229551-istio-podinfo" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:59:45.193489 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.193341    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-kube-api-access-6ph7f" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "kube-api-access-6ph7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:45.193637 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.193563    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-istio-token" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:45.193673 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.193651    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-envoy" pod "3bdf8d45-c633-456c-a6f0-268812229551" (UID: "3bdf8d45-c633-456c-a6f0-268812229551"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:45.295410 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.295374    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdf8d45-c633-456c-a6f0-268812229551-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.295410 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.295410    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ph7f\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-kube-api-access-6ph7f\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.295672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.295421    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdf8d45-c633-456c-a6f0-268812229551-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.295672 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.295430    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdf8d45-c633-456c-a6f0-268812229551-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:45.506068 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.505967    5196 generic.go:356] "Generic (PLEG): container finished" podID="3bdf8d45-c633-456c-a6f0-268812229551" containerID="cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52" exitCode=0
Nov 03 20:59:45.506068 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.506031    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"
Nov 03 20:59:45.506068 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.506056    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" event={"ID":"3bdf8d45-c633-456c-a6f0-268812229551","Type":"ContainerDied","Data":"cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52"}
Nov 03 20:59:45.506413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.506135    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4" event={"ID":"3bdf8d45-c633-456c-a6f0-268812229551","Type":"ContainerDied","Data":"2e021416b8f5f0a7f1f4abd7ec478b165831a54d6845fcfe7c132ca6ecd9d49b"}
Nov 03 20:59:45.506413 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.506165    5196 scope.go:122] "RemoveContainer" containerID="cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52"
Nov 03 20:59:45.524124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.521266    5196 scope.go:122] "RemoveContainer" containerID="cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52"
Nov 03 20:59:45.524124 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:59:45.522119    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52\": container with ID starting with cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52 not found: ID does not exist" containerID="cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52"
Nov 03 20:59:45.524124 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.522155    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52"} err="failed to get container status \"cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52\": rpc error: code = NotFound desc = could not find container \"cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52\": container with ID starting with cf2dad2b3180835991604c9e927c12d8281a3db07dde3d8126c46f40faf5cf52 not found: ID does not exist"
Nov 03 20:59:45.664446 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.664411    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"]
Nov 03 20:59:45.669011 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:45.668982    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-w9bg4"]
Nov 03 20:59:46.098873 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:46.098834    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:46.112579 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:46.112544    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bdf8d45-c633-456c-a6f0-268812229551" path="/var/lib/kubelet/pods/3bdf8d45-c633-456c-a6f0-268812229551/volumes"
Nov 03 20:59:46.273530 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:46.273495    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:46.427177 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:46.426724    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:48.488472 ip-10-0-128-49 kubenswrapper[5196]: E1103 20:59:48.488424    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 20:59:49.864321 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:49.864284    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 20:59:51.121120 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:51.117833    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:54.462853 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:54.462814    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:54.634255 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:54.634209    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:56.046753 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:56.046719    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:56.217916 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:56.214375    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:56.698071 ip-10-0-128-49 kubenswrapper[5196]: I1103 20:59:56.698013    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:00.271121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.264459    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"]
Nov 03 21:00:00.271121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.266685    5196 cpu_manager.go:414] "containerMap: removing container" podUID="23504e58-91da-43d9-a79a-17b69fe84730" containerName="revoke-keys"
Nov 03 21:00:00.271121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.266711    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23504e58-91da-43d9-a79a-17b69fe84730" containerName="revoke-keys"
Nov 03 21:00:00.271121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.266742    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy"
Nov 03 21:00:00.271121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.266751    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy"
Nov 03 21:00:00.271121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.267010    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="23504e58-91da-43d9-a79a-17b69fe84730" containerName="revoke-keys"
Nov 03 21:00:00.271121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.267030    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bdf8d45-c633-456c-a6f0-268812229551" containerName="istio-proxy"
Nov 03 21:00:00.291835 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.291803    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"]
Nov 03 21:00:00.296150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.292163    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:00.299910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.299777    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"]
Nov 03 21:00:00.384149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.299883    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:00.384149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.358434    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f38d87-0424-422b-859a-97a3636b4004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx\" (UID: \"b5f38d87-0424-422b-859a-97a3636b4004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:00.384149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.383600    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0862fa4d-8c1b-4933-bd44-0b370beffe5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5\" (UID: \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:00.384149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.383794    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qc5\" (UniqueName: \"kubernetes.io/projected/0862fa4d-8c1b-4933-bd44-0b370beffe5c-kube-api-access-l8qc5\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5\" (UID: \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:00.384149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.384001    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszxc\" (UniqueName: \"kubernetes.io/projected/b5f38d87-0424-422b-859a-97a3636b4004-kube-api-access-wszxc\") pod \"maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx\" (UID: \"b5f38d87-0424-422b-859a-97a3636b4004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:00.404611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.403704    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"]
Nov 03 21:00:00.404611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.404336    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:00.417649 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.417577    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"]
Nov 03 21:00:00.418971 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.418910    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:00.430945 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.430913    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:00:00.436384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.436354    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"]
Nov 03 21:00:00.438660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.438630    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:00.442474 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.442449    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:00.443072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.443056    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"]
Nov 03 21:00:00.444306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.444283    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:00.448587 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.448572    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"]
Nov 03 21:00:00.449268 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.448672    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:00.455374 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.455348    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"]
Nov 03 21:00:00.455473 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.455395    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"]
Nov 03 21:00:00.455473 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.455422    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"]
Nov 03 21:00:00.455473 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.455446    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"]
Nov 03 21:00:00.455473 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.455470    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"]
Nov 03 21:00:00.455613 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.455492    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"]
Nov 03 21:00:00.455613 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.455580    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:00.461018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.460992    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"]
Nov 03 21:00:00.467416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.467068    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"]
Nov 03 21:00:00.486688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.486656    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2812b4-8772-4e50-a5ff-e09e788fffbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4\" (UID: \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:00.486834 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.486717    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcng8\" (UniqueName: \"kubernetes.io/projected/1e2812b4-8772-4e50-a5ff-e09e788fffbe-kube-api-access-wcng8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4\" (UID: \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:00.486931 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.486903    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0862fa4d-8c1b-4933-bd44-0b370beffe5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5\" (UID: \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:00.487082 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.487061    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qc5\" (UniqueName: \"kubernetes.io/projected/0862fa4d-8c1b-4933-bd44-0b370beffe5c-kube-api-access-l8qc5\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5\" (UID: \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:00.487282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.487251    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszxc\" (UniqueName: \"kubernetes.io/projected/b5f38d87-0424-422b-859a-97a3636b4004-kube-api-access-wszxc\") pod \"maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx\" (UID: \"b5f38d87-0424-422b-859a-97a3636b4004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:00.487397 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.487311    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f38d87-0424-422b-859a-97a3636b4004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx\" (UID: \"b5f38d87-0424-422b-859a-97a3636b4004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:00.487595 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.487577    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0862fa4d-8c1b-4933-bd44-0b370beffe5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5\" (UID: \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:00.488579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.488554    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f38d87-0424-422b-859a-97a3636b4004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx\" (UID: \"b5f38d87-0424-422b-859a-97a3636b4004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:00.541341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.541266    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qc5\" (UniqueName: \"kubernetes.io/projected/0862fa4d-8c1b-4933-bd44-0b370beffe5c-kube-api-access-l8qc5\") pod \"maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5\" (UID: \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:00.541483 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.541379    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszxc\" (UniqueName: \"kubernetes.io/projected/b5f38d87-0424-422b-859a-97a3636b4004-kube-api-access-wszxc\") pod \"maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx\" (UID: \"b5f38d87-0424-422b-859a-97a3636b4004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:00.588635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.588585    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88knz\" (UniqueName: \"kubernetes.io/projected/704419eb-b6b8-4112-9540-fe79b3991af0-kube-api-access-88knz\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh\" (UID: \"704419eb-b6b8-4112-9540-fe79b3991af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:00.588830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.588722    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrbs\" (UniqueName: \"kubernetes.io/projected/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-kube-api-access-wlrbs\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq\" (UID: \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:00.588870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.588849    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwk86\" (UniqueName: \"kubernetes.io/projected/c95f6b0e-1edb-47c8-9ed2-201f135db605-kube-api-access-gwk86\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69\" (UID: \"c95f6b0e-1edb-47c8-9ed2-201f135db605\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:00.588914 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.588875    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2812b4-8772-4e50-a5ff-e09e788fffbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4\" (UID: \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:00.588914 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.588901    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcng8\" (UniqueName: \"kubernetes.io/projected/1e2812b4-8772-4e50-a5ff-e09e788fffbe-kube-api-access-wcng8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4\" (UID: \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:00.588988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.588921    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq\" (UID: \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:00.589076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.589052    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704419eb-b6b8-4112-9540-fe79b3991af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh\" (UID: \"704419eb-b6b8-4112-9540-fe79b3991af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:00.589251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.589219    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxp8d\" (UniqueName: \"kubernetes.io/projected/aaed4e19-f721-4816-af5e-97fe99adde27-kube-api-access-dxp8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g\" (UID: \"aaed4e19-f721-4816-af5e-97fe99adde27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:00.589358 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.589344    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4aead-2076-4cbf-9322-8d472ccf258d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf\" (UID: \"b7c4aead-2076-4cbf-9322-8d472ccf258d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:00.589444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.589420    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvr2l\" (UniqueName: \"kubernetes.io/projected/b7c4aead-2076-4cbf-9322-8d472ccf258d-kube-api-access-zvr2l\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf\" (UID: \"b7c4aead-2076-4cbf-9322-8d472ccf258d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:00.589480 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.589469    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed4e19-f721-4816-af5e-97fe99adde27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g\" (UID: \"aaed4e19-f721-4816-af5e-97fe99adde27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:00.589651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.589631    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95f6b0e-1edb-47c8-9ed2-201f135db605-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69\" (UID: \"c95f6b0e-1edb-47c8-9ed2-201f135db605\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:00.589698 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.589679    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2812b4-8772-4e50-a5ff-e09e788fffbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4\" (UID: \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:00.642909 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.642872    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcng8\" (UniqueName: \"kubernetes.io/projected/1e2812b4-8772-4e50-a5ff-e09e788fffbe-kube-api-access-wcng8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4\" (UID: \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:00.691316 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691274    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrbs\" (UniqueName: \"kubernetes.io/projected/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-kube-api-access-wlrbs\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq\" (UID: \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:00.691491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691375    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwk86\" (UniqueName: \"kubernetes.io/projected/c95f6b0e-1edb-47c8-9ed2-201f135db605-kube-api-access-gwk86\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69\" (UID: \"c95f6b0e-1edb-47c8-9ed2-201f135db605\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:00.691491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691434    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq\" (UID: \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:00.691491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691473    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704419eb-b6b8-4112-9540-fe79b3991af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh\" (UID: \"704419eb-b6b8-4112-9540-fe79b3991af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:00.691640 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691518    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxp8d\" (UniqueName: \"kubernetes.io/projected/aaed4e19-f721-4816-af5e-97fe99adde27-kube-api-access-dxp8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g\" (UID: \"aaed4e19-f721-4816-af5e-97fe99adde27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:00.691640 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691621    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4aead-2076-4cbf-9322-8d472ccf258d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf\" (UID: \"b7c4aead-2076-4cbf-9322-8d472ccf258d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:00.691736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691643    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvr2l\" (UniqueName: \"kubernetes.io/projected/b7c4aead-2076-4cbf-9322-8d472ccf258d-kube-api-access-zvr2l\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf\" (UID: \"b7c4aead-2076-4cbf-9322-8d472ccf258d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:00.691736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691679    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed4e19-f721-4816-af5e-97fe99adde27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g\" (UID: \"aaed4e19-f721-4816-af5e-97fe99adde27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:00.691843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691781    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95f6b0e-1edb-47c8-9ed2-201f135db605-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69\" (UID: \"c95f6b0e-1edb-47c8-9ed2-201f135db605\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:00.691945 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.691909    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88knz\" (UniqueName: \"kubernetes.io/projected/704419eb-b6b8-4112-9540-fe79b3991af0-kube-api-access-88knz\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh\" (UID: \"704419eb-b6b8-4112-9540-fe79b3991af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:00.692389 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.692365    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4aead-2076-4cbf-9322-8d472ccf258d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf\" (UID: \"b7c4aead-2076-4cbf-9322-8d472ccf258d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:00.692469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.692432    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704419eb-b6b8-4112-9540-fe79b3991af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh\" (UID: \"704419eb-b6b8-4112-9540-fe79b3991af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:00.692469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.692450    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed4e19-f721-4816-af5e-97fe99adde27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g\" (UID: \"aaed4e19-f721-4816-af5e-97fe99adde27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:00.692543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.692431    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq\" (UID: \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:00.692650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.692630    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95f6b0e-1edb-47c8-9ed2-201f135db605-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69\" (UID: \"c95f6b0e-1edb-47c8-9ed2-201f135db605\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:00.701054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.701026    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:00.725392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.725358    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:00.738865 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.738825    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:00.748423 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.748388    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxp8d\" (UniqueName: \"kubernetes.io/projected/aaed4e19-f721-4816-af5e-97fe99adde27-kube-api-access-dxp8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g\" (UID: \"aaed4e19-f721-4816-af5e-97fe99adde27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:00.750070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.750037    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvr2l\" (UniqueName: \"kubernetes.io/projected/b7c4aead-2076-4cbf-9322-8d472ccf258d-kube-api-access-zvr2l\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf\" (UID: \"b7c4aead-2076-4cbf-9322-8d472ccf258d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:00.750357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.750335    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88knz\" (UniqueName: \"kubernetes.io/projected/704419eb-b6b8-4112-9540-fe79b3991af0-kube-api-access-88knz\") pod \"maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh\" (UID: \"704419eb-b6b8-4112-9540-fe79b3991af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:00.752600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.752570    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwk86\" (UniqueName: \"kubernetes.io/projected/c95f6b0e-1edb-47c8-9ed2-201f135db605-kube-api-access-gwk86\") pod \"maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69\" (UID: \"c95f6b0e-1edb-47c8-9ed2-201f135db605\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:00.754478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.754448    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrbs\" (UniqueName: \"kubernetes.io/projected/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-kube-api-access-wlrbs\") pod \"maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq\" (UID: \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:00.770080 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.769696    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:00.785463 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.785426    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:00.805742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.800461    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:00.805742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:00.804890    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:01.062411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.058130    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:01.163132 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.124497    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:01.512970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.512939    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"]
Nov 03 21:00:01.517088 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:00:01.517052    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0862fa4d_8c1b_4933_bd44_0b370beffe5c.slice/crio-769a1fbe10f84de31efb7efc4561a70b8d83eca4ae61ac151fae6654473e028d WatchSource:0}: Error finding container 769a1fbe10f84de31efb7efc4561a70b8d83eca4ae61ac151fae6654473e028d: Status 404 returned error can't find the container with id 769a1fbe10f84de31efb7efc4561a70b8d83eca4ae61ac151fae6654473e028d
Nov 03 21:00:01.639275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.639233    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:01.778990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.760845    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"]
Nov 03 21:00:01.778990 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:00:01.764248    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f38d87_0424_422b_859a_97a3636b4004.slice/crio-290d61883965cd8319f0fc6fc3331469ec9e999ba10adaf788cb34e1219b8ae9 WatchSource:0}: Error finding container 290d61883965cd8319f0fc6fc3331469ec9e999ba10adaf788cb34e1219b8ae9: Status 404 returned error can't find the container with id 290d61883965cd8319f0fc6fc3331469ec9e999ba10adaf788cb34e1219b8ae9
Nov 03 21:00:01.778990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.778281    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"]
Nov 03 21:00:01.785643 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:00:01.785600    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95f6b0e_1edb_47c8_9ed2_201f135db605.slice/crio-6fe522b572d12725a114f03773c1715dd29cef2a1528096ab62bf12c2b9c811f WatchSource:0}: Error finding container 6fe522b572d12725a114f03773c1715dd29cef2a1528096ab62bf12c2b9c811f: Status 404 returned error can't find the container with id 6fe522b572d12725a114f03773c1715dd29cef2a1528096ab62bf12c2b9c811f
Nov 03 21:00:01.811168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.811137    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"]
Nov 03 21:00:01.818560 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.818528    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"]
Nov 03 21:00:01.819475 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:00:01.819444    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c4aead_2076_4cbf_9322_8d472ccf258d.slice/crio-24b460cbbff3073f75bcbe8e2987de351aa20db76184e77dc76be1381813aa45 WatchSource:0}: Error finding container 24b460cbbff3073f75bcbe8e2987de351aa20db76184e77dc76be1381813aa45: Status 404 returned error can't find the container with id 24b460cbbff3073f75bcbe8e2987de351aa20db76184e77dc76be1381813aa45
Nov 03 21:00:01.838672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.838312    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"]
Nov 03 21:00:01.880456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.880430    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"]
Nov 03 21:00:01.885492 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.885464    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"]
Nov 03 21:00:01.891960 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:00:01.891930    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704419eb_b6b8_4112_9540_fe79b3991af0.slice/crio-202ae1131e3188bf8430fa73e0b3158747ce73ee8fe0f90d0b11b82582e4a868 WatchSource:0}: Error finding container 202ae1131e3188bf8430fa73e0b3158747ce73ee8fe0f90d0b11b82582e4a868: Status 404 returned error can't find the container with id 202ae1131e3188bf8430fa73e0b3158747ce73ee8fe0f90d0b11b82582e4a868
Nov 03 21:00:01.897901 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.897717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:01.935175 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.935058    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" event={"ID":"704419eb-b6b8-4112-9540-fe79b3991af0","Type":"ContainerStarted","Data":"202ae1131e3188bf8430fa73e0b3158747ce73ee8fe0f90d0b11b82582e4a868"}
Nov 03 21:00:01.940326 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.940293    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" event={"ID":"99a0f4a8-3053-4e62-86ce-68d0cd531ab5","Type":"ContainerStarted","Data":"e24c9bc34dfc199d690d2968be1d25c77b8d20ff0aeab1aa1b05979250277876"}
Nov 03 21:00:01.942109 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.942068    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" event={"ID":"1e2812b4-8772-4e50-a5ff-e09e788fffbe","Type":"ContainerStarted","Data":"949de73a5ce9855e22883e7dd0a8e187653c590805e5c77882110cdf7043c1a2"}
Nov 03 21:00:01.960732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.960688    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" event={"ID":"c95f6b0e-1edb-47c8-9ed2-201f135db605","Type":"ContainerStarted","Data":"6fe522b572d12725a114f03773c1715dd29cef2a1528096ab62bf12c2b9c811f"}
Nov 03 21:00:01.963582 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.963552    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" event={"ID":"aaed4e19-f721-4816-af5e-97fe99adde27","Type":"ContainerStarted","Data":"8154578596ac280a47342bf2290c5b0d8d4fd4decf0493e374761b0ecaba6e69"}
Nov 03 21:00:01.965966 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.965924    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" event={"ID":"b7c4aead-2076-4cbf-9322-8d472ccf258d","Type":"ContainerStarted","Data":"24b460cbbff3073f75bcbe8e2987de351aa20db76184e77dc76be1381813aa45"}
Nov 03 21:00:01.967648 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.967617    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" event={"ID":"b5f38d87-0424-422b-859a-97a3636b4004","Type":"ContainerStarted","Data":"290d61883965cd8319f0fc6fc3331469ec9e999ba10adaf788cb34e1219b8ae9"}
Nov 03 21:00:01.969065 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:01.969040    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" event={"ID":"0862fa4d-8c1b-4933-bd44-0b370beffe5c","Type":"ContainerStarted","Data":"769a1fbe10f84de31efb7efc4561a70b8d83eca4ae61ac151fae6654473e028d"}
Nov 03 21:00:02.982508 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.982463    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" event={"ID":"704419eb-b6b8-4112-9540-fe79b3991af0","Type":"ContainerStarted","Data":"c4c597bf21135ce95fb801a41c93c6df934b32e8a778a4cc42fade8f0fd5aca7"}
Nov 03 21:00:02.984384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.984356    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" event={"ID":"99a0f4a8-3053-4e62-86ce-68d0cd531ab5","Type":"ContainerStarted","Data":"3f999d23fba2283de4cb6311459446e1852f63e371e9f5c4835eae3b9796fd5e"}
Nov 03 21:00:02.986192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.986163    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" event={"ID":"1e2812b4-8772-4e50-a5ff-e09e788fffbe","Type":"ContainerStarted","Data":"80a8158aa4d8eb7935f63e5e395c3907d8e8ba12d18512b4582990fe326f8c68"}
Nov 03 21:00:02.987818 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.987792    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" event={"ID":"c95f6b0e-1edb-47c8-9ed2-201f135db605","Type":"ContainerStarted","Data":"67295bfb2e6f936516a7955e66266d7e5f7a8902e241f9c81ad0116b70bf8ea3"}
Nov 03 21:00:02.989696 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.989667    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" event={"ID":"aaed4e19-f721-4816-af5e-97fe99adde27","Type":"ContainerStarted","Data":"2d3f60c4b3eeb2ad615196cfe7a271dba4cdcb50dd1b273694414a0aafea91a0"}
Nov 03 21:00:02.991242 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.991216    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" event={"ID":"b7c4aead-2076-4cbf-9322-8d472ccf258d","Type":"ContainerStarted","Data":"788a62f11d4602513095cafc795333d03bbbff7cc39cdd3f30f60459e4b6d6e4"}
Nov 03 21:00:02.993020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.992998    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" event={"ID":"b5f38d87-0424-422b-859a-97a3636b4004","Type":"ContainerStarted","Data":"a0099928c46a383be2bf5f2ca17c70af96ddfaaa939f2f8ec23d5027da343b98"}
Nov 03 21:00:02.994658 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.994635    5196 generic.go:356] "Generic (PLEG): container finished" podID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerID="467913c9dd6da916cfffc9d20e02bea00d7a01f5a9fd323bc19b14e6892597d7" exitCode=60
Nov 03 21:00:02.994763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.994712    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" event={"ID":"0862fa4d-8c1b-4933-bd44-0b370beffe5c","Type":"ContainerDied","Data":"467913c9dd6da916cfffc9d20e02bea00d7a01f5a9fd323bc19b14e6892597d7"}
Nov 03 21:00:02.994886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:02.994872    5196 scope.go:122] "RemoveContainer" containerID="467913c9dd6da916cfffc9d20e02bea00d7a01f5a9fd323bc19b14e6892597d7"
Nov 03 21:00:03.093765 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:03.093709    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" podStartSLOduration=3.093693795 podStartE2EDuration="3.093693795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.088612529 +0000 UTC m=+1843.092571267" watchObservedRunningTime="2026-11-03 21:00:03.093693795 +0000 UTC m=+1843.097652519"
Nov 03 21:00:03.197112 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:03.197002    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" podStartSLOduration=3.196987409 podStartE2EDuration="3.196987409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.196007898 +0000 UTC m=+1843.199966640" watchObservedRunningTime="2026-11-03 21:00:03.196987409 +0000 UTC m=+1843.200946134"
Nov 03 21:00:03.307875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:03.307822    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" podStartSLOduration=3.307807727 podStartE2EDuration="3.307807727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.304246449 +0000 UTC m=+1843.308205173" watchObservedRunningTime="2026-11-03 21:00:03.307807727 +0000 UTC m=+1843.311766450"
Nov 03 21:00:03.415137 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:03.415036    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" podStartSLOduration=3.415022195 podStartE2EDuration="3.415022195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.410813003 +0000 UTC m=+1843.414771728" watchObservedRunningTime="2026-11-03 21:00:03.415022195 +0000 UTC m=+1843.418980918"
Nov 03 21:00:03.631150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:03.631060    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" podStartSLOduration=3.631038412 podStartE2EDuration="3.631038412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.62549446 +0000 UTC m=+1843.629453183" watchObservedRunningTime="2026-11-03 21:00:03.631038412 +0000 UTC m=+1843.634997131"
Nov 03 21:00:03.734194 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:03.734061    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" podStartSLOduration=3.734046409 podStartE2EDuration="3.734046409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.731324179 +0000 UTC m=+1843.735282917" watchObservedRunningTime="2026-11-03 21:00:03.734046409 +0000 UTC m=+1843.738005129"
Nov 03 21:00:04.036286 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.036180    5196 generic.go:356] "Generic (PLEG): container finished" podID="b5f38d87-0424-422b-859a-97a3636b4004" containerID="a0099928c46a383be2bf5f2ca17c70af96ddfaaa939f2f8ec23d5027da343b98" exitCode=60
Nov 03 21:00:04.036931 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.036320    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" event={"ID":"b5f38d87-0424-422b-859a-97a3636b4004","Type":"ContainerDied","Data":"a0099928c46a383be2bf5f2ca17c70af96ddfaaa939f2f8ec23d5027da343b98"}
Nov 03 21:00:04.037406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.037382    5196 scope.go:122] "RemoveContainer" containerID="a0099928c46a383be2bf5f2ca17c70af96ddfaaa939f2f8ec23d5027da343b98"
Nov 03 21:00:04.040177 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.040130    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" event={"ID":"0862fa4d-8c1b-4933-bd44-0b370beffe5c","Type":"ContainerStarted","Data":"ca7668e1960b70b98ba849e4c42d9c35ddc0165b393edae9a5a382270356ce98"}
Nov 03 21:00:04.042465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.042440    5196 generic.go:356] "Generic (PLEG): container finished" podID="704419eb-b6b8-4112-9540-fe79b3991af0" containerID="c4c597bf21135ce95fb801a41c93c6df934b32e8a778a4cc42fade8f0fd5aca7" exitCode=60
Nov 03 21:00:04.042556 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.042525    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" event={"ID":"704419eb-b6b8-4112-9540-fe79b3991af0","Type":"ContainerDied","Data":"c4c597bf21135ce95fb801a41c93c6df934b32e8a778a4cc42fade8f0fd5aca7"}
Nov 03 21:00:04.042837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.042821    5196 scope.go:122] "RemoveContainer" containerID="c4c597bf21135ce95fb801a41c93c6df934b32e8a778a4cc42fade8f0fd5aca7"
Nov 03 21:00:04.044849 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.044811    5196 generic.go:356] "Generic (PLEG): container finished" podID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerID="3f999d23fba2283de4cb6311459446e1852f63e371e9f5c4835eae3b9796fd5e" exitCode=60
Nov 03 21:00:04.044946 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.044924    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" event={"ID":"99a0f4a8-3053-4e62-86ce-68d0cd531ab5","Type":"ContainerDied","Data":"3f999d23fba2283de4cb6311459446e1852f63e371e9f5c4835eae3b9796fd5e"}
Nov 03 21:00:04.045381 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.045359    5196 scope.go:122] "RemoveContainer" containerID="3f999d23fba2283de4cb6311459446e1852f63e371e9f5c4835eae3b9796fd5e"
Nov 03 21:00:04.047314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.047288    5196 generic.go:356] "Generic (PLEG): container finished" podID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerID="80a8158aa4d8eb7935f63e5e395c3907d8e8ba12d18512b4582990fe326f8c68" exitCode=60
Nov 03 21:00:04.047429 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.047365    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" event={"ID":"1e2812b4-8772-4e50-a5ff-e09e788fffbe","Type":"ContainerDied","Data":"80a8158aa4d8eb7935f63e5e395c3907d8e8ba12d18512b4582990fe326f8c68"}
Nov 03 21:00:04.047626 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.047612    5196 scope.go:122] "RemoveContainer" containerID="80a8158aa4d8eb7935f63e5e395c3907d8e8ba12d18512b4582990fe326f8c68"
Nov 03 21:00:04.049444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.049409    5196 generic.go:356] "Generic (PLEG): container finished" podID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerID="67295bfb2e6f936516a7955e66266d7e5f7a8902e241f9c81ad0116b70bf8ea3" exitCode=60
Nov 03 21:00:04.049544 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.049493    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" event={"ID":"c95f6b0e-1edb-47c8-9ed2-201f135db605","Type":"ContainerDied","Data":"67295bfb2e6f936516a7955e66266d7e5f7a8902e241f9c81ad0116b70bf8ea3"}
Nov 03 21:00:04.049778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.049756    5196 scope.go:122] "RemoveContainer" containerID="67295bfb2e6f936516a7955e66266d7e5f7a8902e241f9c81ad0116b70bf8ea3"
Nov 03 21:00:04.051491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.051467    5196 generic.go:356] "Generic (PLEG): container finished" podID="aaed4e19-f721-4816-af5e-97fe99adde27" containerID="2d3f60c4b3eeb2ad615196cfe7a271dba4cdcb50dd1b273694414a0aafea91a0" exitCode=60
Nov 03 21:00:04.051659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.051634    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" event={"ID":"aaed4e19-f721-4816-af5e-97fe99adde27","Type":"ContainerDied","Data":"2d3f60c4b3eeb2ad615196cfe7a271dba4cdcb50dd1b273694414a0aafea91a0"}
Nov 03 21:00:04.052013 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.051994    5196 scope.go:122] "RemoveContainer" containerID="2d3f60c4b3eeb2ad615196cfe7a271dba4cdcb50dd1b273694414a0aafea91a0"
Nov 03 21:00:04.055049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.054276    5196 generic.go:356] "Generic (PLEG): container finished" podID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerID="788a62f11d4602513095cafc795333d03bbbff7cc39cdd3f30f60459e4b6d6e4" exitCode=60
Nov 03 21:00:04.055049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.054337    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" event={"ID":"b7c4aead-2076-4cbf-9322-8d472ccf258d","Type":"ContainerDied","Data":"788a62f11d4602513095cafc795333d03bbbff7cc39cdd3f30f60459e4b6d6e4"}
Nov 03 21:00:04.055049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.054698    5196 scope.go:122] "RemoveContainer" containerID="788a62f11d4602513095cafc795333d03bbbff7cc39cdd3f30f60459e4b6d6e4"
Nov 03 21:00:04.152461 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.152388    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" podStartSLOduration=4.152366852 podStartE2EDuration="4.152366852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.843430351 +0000 UTC m=+1843.847389180" watchObservedRunningTime="2026-11-03 21:00:04.152366852 +0000 UTC m=+1844.156325577"
Nov 03 21:00:04.573170 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.573062    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" podStartSLOduration=4.573040697 podStartE2EDuration="4.573040697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:04.571140883 +0000 UTC m=+1844.575099625" watchObservedRunningTime="2026-11-03 21:00:04.573040697 +0000 UTC m=+1844.576999421"
Nov 03 21:00:04.896280 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:04.896234    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:05.066611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.066506    5196 generic.go:356] "Generic (PLEG): container finished" podID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerID="ca7668e1960b70b98ba849e4c42d9c35ddc0165b393edae9a5a382270356ce98" exitCode=60
Nov 03 21:00:05.066611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.066573    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" event={"ID":"0862fa4d-8c1b-4933-bd44-0b370beffe5c","Type":"ContainerDied","Data":"ca7668e1960b70b98ba849e4c42d9c35ddc0165b393edae9a5a382270356ce98"}
Nov 03 21:00:05.067187 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.066682    5196 scope.go:122] "RemoveContainer" containerID="467913c9dd6da916cfffc9d20e02bea00d7a01f5a9fd323bc19b14e6892597d7"
Nov 03 21:00:05.067187 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.066938    5196 scope.go:122] "RemoveContainer" containerID="ca7668e1960b70b98ba849e4c42d9c35ddc0165b393edae9a5a382270356ce98"
Nov 03 21:00:05.067496 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:05.067470    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5_odh-ai-gateway-infra(0862fa4d-8c1b-4933-bd44-0b370beffe5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c"
Nov 03 21:00:05.069154 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.069126    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" event={"ID":"704419eb-b6b8-4112-9540-fe79b3991af0","Type":"ContainerStarted","Data":"6b57a5b16cb111dba69aca915ada76a89254bab43d1a070a18614a7db1b56520"}
Nov 03 21:00:05.071347 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.071324    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" event={"ID":"99a0f4a8-3053-4e62-86ce-68d0cd531ab5","Type":"ContainerStarted","Data":"48f6d7c35d5bb9234a8b04ebfd9b7e4c69d9846ebff4f7395e69121c056ecd63"}
Nov 03 21:00:05.073582 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.073559    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" event={"ID":"1e2812b4-8772-4e50-a5ff-e09e788fffbe","Type":"ContainerStarted","Data":"a82286f803dc5bbe42de2083af49a2bec60310c636c88315281fb6f1e5a6122f"}
Nov 03 21:00:05.075647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.075620    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" event={"ID":"c95f6b0e-1edb-47c8-9ed2-201f135db605","Type":"ContainerStarted","Data":"49c9b71a058711110fbf70f422b90905014f52101e6594176beac87c952d553f"}
Nov 03 21:00:05.077802 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.077780    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" event={"ID":"aaed4e19-f721-4816-af5e-97fe99adde27","Type":"ContainerStarted","Data":"24a009c629e98feb20dc3431a4bf116036ed8af33c68371f72d489ccd6c46b7a"}
Nov 03 21:00:05.080015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.079990    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" event={"ID":"b7c4aead-2076-4cbf-9322-8d472ccf258d","Type":"ContainerStarted","Data":"42ed97163adfe426e920b961ed8d2541f09a7731be2823bee899652ad74cad59"}
Nov 03 21:00:05.082275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:05.082087    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" event={"ID":"b5f38d87-0424-422b-859a-97a3636b4004","Type":"ContainerStarted","Data":"d10b0d076cda365206aee07a39a722b9726d8ec8aad90f69a6f6f34f0dba8bae"}
Nov 03 21:00:06.093855 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.093814    5196 generic.go:356] "Generic (PLEG): container finished" podID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerID="48f6d7c35d5bb9234a8b04ebfd9b7e4c69d9846ebff4f7395e69121c056ecd63" exitCode=60
Nov 03 21:00:06.094424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.093897    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" event={"ID":"99a0f4a8-3053-4e62-86ce-68d0cd531ab5","Type":"ContainerDied","Data":"48f6d7c35d5bb9234a8b04ebfd9b7e4c69d9846ebff4f7395e69121c056ecd63"}
Nov 03 21:00:06.094424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.093957    5196 scope.go:122] "RemoveContainer" containerID="3f999d23fba2283de4cb6311459446e1852f63e371e9f5c4835eae3b9796fd5e"
Nov 03 21:00:06.094424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.094380    5196 scope.go:122] "RemoveContainer" containerID="48f6d7c35d5bb9234a8b04ebfd9b7e4c69d9846ebff4f7395e69121c056ecd63"
Nov 03 21:00:06.094787 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:06.094765    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq_odh-ai-gateway-infra(99a0f4a8-3053-4e62-86ce-68d0cd531ab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5"
Nov 03 21:00:06.096240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.096214    5196 generic.go:356] "Generic (PLEG): container finished" podID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerID="a82286f803dc5bbe42de2083af49a2bec60310c636c88315281fb6f1e5a6122f" exitCode=60
Nov 03 21:00:06.096344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.096297    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" event={"ID":"1e2812b4-8772-4e50-a5ff-e09e788fffbe","Type":"ContainerDied","Data":"a82286f803dc5bbe42de2083af49a2bec60310c636c88315281fb6f1e5a6122f"}
Nov 03 21:00:06.096634 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.096620    5196 scope.go:122] "RemoveContainer" containerID="a82286f803dc5bbe42de2083af49a2bec60310c636c88315281fb6f1e5a6122f"
Nov 03 21:00:06.096993 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:06.096973    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4_odh-ai-gateway-infra(1e2812b4-8772-4e50-a5ff-e09e788fffbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe"
Nov 03 21:00:06.098367 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.098345    5196 generic.go:356] "Generic (PLEG): container finished" podID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerID="49c9b71a058711110fbf70f422b90905014f52101e6594176beac87c952d553f" exitCode=60
Nov 03 21:00:06.098481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.098410    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" event={"ID":"c95f6b0e-1edb-47c8-9ed2-201f135db605","Type":"ContainerDied","Data":"49c9b71a058711110fbf70f422b90905014f52101e6594176beac87c952d553f"}
Nov 03 21:00:06.098756 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.098735    5196 scope.go:122] "RemoveContainer" containerID="49c9b71a058711110fbf70f422b90905014f52101e6594176beac87c952d553f"
Nov 03 21:00:06.099140 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:06.099081    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69_odh-ai-gateway-infra(c95f6b0e-1edb-47c8-9ed2-201f135db605)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605"
Nov 03 21:00:06.100857 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.100837    5196 generic.go:356] "Generic (PLEG): container finished" podID="aaed4e19-f721-4816-af5e-97fe99adde27" containerID="24a009c629e98feb20dc3431a4bf116036ed8af33c68371f72d489ccd6c46b7a" exitCode=60
Nov 03 21:00:06.101334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.101315    5196 scope.go:122] "RemoveContainer" containerID="24a009c629e98feb20dc3431a4bf116036ed8af33c68371f72d489ccd6c46b7a"
Nov 03 21:00:06.101708 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:06.101673    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g_odh-ai-gateway-infra(aaed4e19-f721-4816-af5e-97fe99adde27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" podUID="aaed4e19-f721-4816-af5e-97fe99adde27"
Nov 03 21:00:06.103262 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.103237    5196 generic.go:356] "Generic (PLEG): container finished" podID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerID="42ed97163adfe426e920b961ed8d2541f09a7731be2823bee899652ad74cad59" exitCode=60
Nov 03 21:00:06.103644 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.103625    5196 scope.go:122] "RemoveContainer" containerID="42ed97163adfe426e920b961ed8d2541f09a7731be2823bee899652ad74cad59"
Nov 03 21:00:06.104004 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:06.103982    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf_odh-ai-gateway-infra(b7c4aead-2076-4cbf-9322-8d472ccf258d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d"
Nov 03 21:00:06.106030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.106008    5196 generic.go:356] "Generic (PLEG): container finished" podID="b5f38d87-0424-422b-859a-97a3636b4004" containerID="d10b0d076cda365206aee07a39a722b9726d8ec8aad90f69a6f6f34f0dba8bae" exitCode=60
Nov 03 21:00:06.106459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.106437    5196 scope.go:122] "RemoveContainer" containerID="d10b0d076cda365206aee07a39a722b9726d8ec8aad90f69a6f6f34f0dba8bae"
Nov 03 21:00:06.106766 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:06.106742    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx_odh-ai-gateway-infra(b5f38d87-0424-422b-859a-97a3636b4004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" podUID="b5f38d87-0424-422b-859a-97a3636b4004"
Nov 03 21:00:06.108460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.108443    5196 scope.go:122] "RemoveContainer" containerID="ca7668e1960b70b98ba849e4c42d9c35ddc0165b393edae9a5a382270356ce98"
Nov 03 21:00:06.108734 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:06.108712    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5_odh-ai-gateway-infra(0862fa4d-8c1b-4933-bd44-0b370beffe5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c"
Nov 03 21:00:06.110581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.110558    5196 generic.go:356] "Generic (PLEG): container finished" podID="704419eb-b6b8-4112-9540-fe79b3991af0" containerID="6b57a5b16cb111dba69aca915ada76a89254bab43d1a070a18614a7db1b56520" exitCode=60
Nov 03 21:00:06.110865 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.110851    5196 scope.go:122] "RemoveContainer" containerID="6b57a5b16cb111dba69aca915ada76a89254bab43d1a070a18614a7db1b56520"
Nov 03 21:00:06.111195 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:06.111176    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh_odh-ai-gateway-infra(704419eb-b6b8-4112-9540-fe79b3991af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" podUID="704419eb-b6b8-4112-9540-fe79b3991af0"
Nov 03 21:00:06.111596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.111572    5196 scope.go:122] "RemoveContainer" containerID="80a8158aa4d8eb7935f63e5e395c3907d8e8ba12d18512b4582990fe326f8c68"
Nov 03 21:00:06.114102 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.114064    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" event={"ID":"aaed4e19-f721-4816-af5e-97fe99adde27","Type":"ContainerDied","Data":"24a009c629e98feb20dc3431a4bf116036ed8af33c68371f72d489ccd6c46b7a"}
Nov 03 21:00:06.114179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.114138    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" event={"ID":"b7c4aead-2076-4cbf-9322-8d472ccf258d","Type":"ContainerDied","Data":"42ed97163adfe426e920b961ed8d2541f09a7731be2823bee899652ad74cad59"}
Nov 03 21:00:06.114179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.114162    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" event={"ID":"b5f38d87-0424-422b-859a-97a3636b4004","Type":"ContainerDied","Data":"d10b0d076cda365206aee07a39a722b9726d8ec8aad90f69a6f6f34f0dba8bae"}
Nov 03 21:00:06.114264 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.114183    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" event={"ID":"704419eb-b6b8-4112-9540-fe79b3991af0","Type":"ContainerDied","Data":"6b57a5b16cb111dba69aca915ada76a89254bab43d1a070a18614a7db1b56520"}
Nov 03 21:00:06.127475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.127448    5196 scope.go:122] "RemoveContainer" containerID="67295bfb2e6f936516a7955e66266d7e5f7a8902e241f9c81ad0116b70bf8ea3"
Nov 03 21:00:06.144755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.144733    5196 scope.go:122] "RemoveContainer" containerID="2d3f60c4b3eeb2ad615196cfe7a271dba4cdcb50dd1b273694414a0aafea91a0"
Nov 03 21:00:06.161928 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.161909    5196 scope.go:122] "RemoveContainer" containerID="788a62f11d4602513095cafc795333d03bbbff7cc39cdd3f30f60459e4b6d6e4"
Nov 03 21:00:06.177731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.177700    5196 scope.go:122] "RemoveContainer" containerID="a0099928c46a383be2bf5f2ca17c70af96ddfaaa939f2f8ec23d5027da343b98"
Nov 03 21:00:06.194828 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.194802    5196 scope.go:122] "RemoveContainer" containerID="c4c597bf21135ce95fb801a41c93c6df934b32e8a778a4cc42fade8f0fd5aca7"
Nov 03 21:00:06.586906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.586857    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:06.787539 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.787501    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:06.947758 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:06.947633    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:07.125017 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:07.124989    5196 scope.go:122] "RemoveContainer" containerID="48f6d7c35d5bb9234a8b04ebfd9b7e4c69d9846ebff4f7395e69121c056ecd63"
Nov 03 21:00:07.125530 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:07.125460    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq_odh-ai-gateway-infra(99a0f4a8-3053-4e62-86ce-68d0cd531ab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5"
Nov 03 21:00:07.126798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:07.126779    5196 scope.go:122] "RemoveContainer" containerID="a82286f803dc5bbe42de2083af49a2bec60310c636c88315281fb6f1e5a6122f"
Nov 03 21:00:07.127216 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:07.127189    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4_odh-ai-gateway-infra(1e2812b4-8772-4e50-a5ff-e09e788fffbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe"
Nov 03 21:00:07.128663 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:07.128645    5196 scope.go:122] "RemoveContainer" containerID="49c9b71a058711110fbf70f422b90905014f52101e6594176beac87c952d553f"
Nov 03 21:00:07.128962 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:07.128932    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69_odh-ai-gateway-infra(c95f6b0e-1edb-47c8-9ed2-201f135db605)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605"
Nov 03 21:00:07.130269 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:07.130252    5196 scope.go:122] "RemoveContainer" containerID="24a009c629e98feb20dc3431a4bf116036ed8af33c68371f72d489ccd6c46b7a"
Nov 03 21:00:07.130532 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:07.130513    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g_odh-ai-gateway-infra(aaed4e19-f721-4816-af5e-97fe99adde27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" podUID="aaed4e19-f721-4816-af5e-97fe99adde27"
Nov 03 21:00:07.132173 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:07.132153    5196 scope.go:122] "RemoveContainer" containerID="42ed97163adfe426e920b961ed8d2541f09a7731be2823bee899652ad74cad59"
Nov 03 21:00:07.132504 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:07.132468    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf_odh-ai-gateway-infra(b7c4aead-2076-4cbf-9322-8d472ccf258d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d"
Nov 03 21:00:07.134028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:07.134009    5196 scope.go:122] "RemoveContainer" containerID="d10b0d076cda365206aee07a39a722b9726d8ec8aad90f69a6f6f34f0dba8bae"
Nov 03 21:00:07.134310 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:07.134287    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx_odh-ai-gateway-infra(b5f38d87-0424-422b-859a-97a3636b4004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" podUID="b5f38d87-0424-422b-859a-97a3636b4004"
Nov 03 21:00:07.135773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:07.135756    5196 scope.go:122] "RemoveContainer" containerID="6b57a5b16cb111dba69aca915ada76a89254bab43d1a070a18614a7db1b56520"
Nov 03 21:00:07.135994 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:07.135977    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh_odh-ai-gateway-infra(704419eb-b6b8-4112-9540-fe79b3991af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" podUID="704419eb-b6b8-4112-9540-fe79b3991af0"
Nov 03 21:00:09.944965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:09.944930    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:10.126934 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:10.126788    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:11.425213 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:11.425179    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:11.629024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:11.628980    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:11.787803 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:11.787706    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:11.963129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:11.961447    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:12.157120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:12.149074    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:16.045979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:16.045943    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"]
Nov 03 21:00:16.092358 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:16.092317    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b45d03-r9m7h"]
Nov 03 21:00:16.111596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:16.111562    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ff596d-0781-4bce-b686-15442140f7dd" path="/var/lib/kubelet/pods/34ff596d-0781-4bce-b686-15442140f7dd/volumes"
Nov 03 21:00:16.112580 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:16.112557    5196 scope.go:122] "RemoveContainer" containerID="a82286f803dc5bbe42de2083af49a2bec60310c636c88315281fb6f1e5a6122f"
Nov 03 21:00:17.101892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:17.101838    5196 scope.go:122] "RemoveContainer" containerID="ca7668e1960b70b98ba849e4c42d9c35ddc0165b393edae9a5a382270356ce98"
Nov 03 21:00:17.173814 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:17.173784    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:17.363119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:17.362259    5196 generic.go:356] "Generic (PLEG): container finished" podID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerID="36c350a21342a37f614983703219765ffbfd18a72c2988e4fa4b4d8cb116e8f3" exitCode=60
Nov 03 21:00:17.363119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:17.362669    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" event={"ID":"1e2812b4-8772-4e50-a5ff-e09e788fffbe","Type":"ContainerDied","Data":"36c350a21342a37f614983703219765ffbfd18a72c2988e4fa4b4d8cb116e8f3"}
Nov 03 21:00:17.363119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:17.362767    5196 scope.go:122] "RemoveContainer" containerID="a82286f803dc5bbe42de2083af49a2bec60310c636c88315281fb6f1e5a6122f"
Nov 03 21:00:17.363448 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:17.363410    5196 scope.go:122] "RemoveContainer" containerID="36c350a21342a37f614983703219765ffbfd18a72c2988e4fa4b4d8cb116e8f3"
Nov 03 21:00:17.382116 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:17.363972    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4_odh-ai-gateway-infra(1e2812b4-8772-4e50-a5ff-e09e788fffbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe"
Nov 03 21:00:17.408120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:17.400424    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:17.552785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:17.552749    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:18.101777 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.101741    5196 scope.go:122] "RemoveContainer" containerID="48f6d7c35d5bb9234a8b04ebfd9b7e4c69d9846ebff4f7395e69121c056ecd63"
Nov 03 21:00:18.101984 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.101900    5196 scope.go:122] "RemoveContainer" containerID="24a009c629e98feb20dc3431a4bf116036ed8af33c68371f72d489ccd6c46b7a"
Nov 03 21:00:18.375633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.375542    5196 generic.go:356] "Generic (PLEG): container finished" podID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerID="25618809643f08d5362bd9c251ba8eea45fbae3c6d8e0b76a1d70804fb6c240e" exitCode=60
Nov 03 21:00:18.375633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.375598    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" event={"ID":"0862fa4d-8c1b-4933-bd44-0b370beffe5c","Type":"ContainerDied","Data":"25618809643f08d5362bd9c251ba8eea45fbae3c6d8e0b76a1d70804fb6c240e"}
Nov 03 21:00:18.375872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.375650    5196 scope.go:122] "RemoveContainer" containerID="ca7668e1960b70b98ba849e4c42d9c35ddc0165b393edae9a5a382270356ce98"
Nov 03 21:00:18.376216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.376199    5196 scope.go:122] "RemoveContainer" containerID="25618809643f08d5362bd9c251ba8eea45fbae3c6d8e0b76a1d70804fb6c240e"
Nov 03 21:00:18.376736 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:18.376665    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5_odh-ai-gateway-infra(0862fa4d-8c1b-4933-bd44-0b370beffe5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c"
Nov 03 21:00:18.487875 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:18.487831    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:00:18.495817 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.495778    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"]
Nov 03 21:00:18.779729 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.779698    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:18.839597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.839567    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2812b4-8772-4e50-a5ff-e09e788fffbe-kube-api-access-wcng8\" (UniqueName: \"kubernetes.io/projected/1e2812b4-8772-4e50-a5ff-e09e788fffbe-kube-api-access-wcng8\") pod \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\" (UID: \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\") "
Nov 03 21:00:18.839790 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.839713    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2812b4-8772-4e50-a5ff-e09e788fffbe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2812b4-8772-4e50-a5ff-e09e788fffbe-maas-api-service-ca\") pod \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\" (UID: \"1e2812b4-8772-4e50-a5ff-e09e788fffbe\") "
Nov 03 21:00:18.840147 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.840073    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2812b4-8772-4e50-a5ff-e09e788fffbe-maas-api-service-ca" pod "1e2812b4-8772-4e50-a5ff-e09e788fffbe" (UID: "1e2812b4-8772-4e50-a5ff-e09e788fffbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:18.840474 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.840443    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2812b4-8772-4e50-a5ff-e09e788fffbe-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:18.841953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.841921    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2812b4-8772-4e50-a5ff-e09e788fffbe-kube-api-access-wcng8" pod "1e2812b4-8772-4e50-a5ff-e09e788fffbe" (UID: "1e2812b4-8772-4e50-a5ff-e09e788fffbe"). InnerVolumeSpecName "kube-api-access-wcng8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:18.941810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:18.941717    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcng8\" (UniqueName: \"kubernetes.io/projected/1e2812b4-8772-4e50-a5ff-e09e788fffbe-kube-api-access-wcng8\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.101690 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.101655    5196 scope.go:122] "RemoveContainer" containerID="d10b0d076cda365206aee07a39a722b9726d8ec8aad90f69a6f6f34f0dba8bae"
Nov 03 21:00:19.101869 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.101839    5196 scope.go:122] "RemoveContainer" containerID="6b57a5b16cb111dba69aca915ada76a89254bab43d1a070a18614a7db1b56520"
Nov 03 21:00:19.391081 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.391044    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" event={"ID":"99a0f4a8-3053-4e62-86ce-68d0cd531ab5","Type":"ContainerStarted","Data":"71780dfefcee52c8050bc466e9b2bd65436506ff41aff4990aa2b10792070b47"}
Nov 03 21:00:19.392691 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.392671    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"
Nov 03 21:00:19.392691 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.392687    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4" event={"ID":"1e2812b4-8772-4e50-a5ff-e09e788fffbe","Type":"ContainerDied","Data":"949de73a5ce9855e22883e7dd0a8e187653c590805e5c77882110cdf7043c1a2"}
Nov 03 21:00:19.392882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.392746    5196 scope.go:122] "RemoveContainer" containerID="36c350a21342a37f614983703219765ffbfd18a72c2988e4fa4b4d8cb116e8f3"
Nov 03 21:00:19.395163 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.395137    5196 generic.go:356] "Generic (PLEG): container finished" podID="aaed4e19-f721-4816-af5e-97fe99adde27" containerID="b940f5012f2218adfeed103fac96c5ff5d78130eb7e2771e42910efa12a1bd84" exitCode=60
Nov 03 21:00:19.395255 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.395178    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" event={"ID":"aaed4e19-f721-4816-af5e-97fe99adde27","Type":"ContainerDied","Data":"b940f5012f2218adfeed103fac96c5ff5d78130eb7e2771e42910efa12a1bd84"}
Nov 03 21:00:19.395581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.395567    5196 scope.go:122] "RemoveContainer" containerID="b940f5012f2218adfeed103fac96c5ff5d78130eb7e2771e42910efa12a1bd84"
Nov 03 21:00:19.395865 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:19.395848    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g_odh-ai-gateway-infra(aaed4e19-f721-4816-af5e-97fe99adde27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" podUID="aaed4e19-f721-4816-af5e-97fe99adde27"
Nov 03 21:00:19.428721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.428695    5196 scope.go:122] "RemoveContainer" containerID="24a009c629e98feb20dc3431a4bf116036ed8af33c68371f72d489ccd6c46b7a"
Nov 03 21:00:19.494997 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.494958    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"]
Nov 03 21:00:19.731392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.731355    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"]
Nov 03 21:00:19.736633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.736591    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-45f792-29895660-fk4t4"]
Nov 03 21:00:19.752518 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.752489    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:19.859225 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.859053    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0862fa4d-8c1b-4933-bd44-0b370beffe5c-kube-api-access-l8qc5\" (UniqueName: \"kubernetes.io/projected/0862fa4d-8c1b-4933-bd44-0b370beffe5c-kube-api-access-l8qc5\") pod \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\" (UID: \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\") "
Nov 03 21:00:19.859425 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.859307    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0862fa4d-8c1b-4933-bd44-0b370beffe5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0862fa4d-8c1b-4933-bd44-0b370beffe5c-maas-api-service-ca\") pod \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\" (UID: \"0862fa4d-8c1b-4933-bd44-0b370beffe5c\") "
Nov 03 21:00:19.859756 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.859727    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0862fa4d-8c1b-4933-bd44-0b370beffe5c-maas-api-service-ca" pod "0862fa4d-8c1b-4933-bd44-0b370beffe5c" (UID: "0862fa4d-8c1b-4933-bd44-0b370beffe5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:19.860265 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.860233    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0862fa4d-8c1b-4933-bd44-0b370beffe5c-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.868296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.867723    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0862fa4d-8c1b-4933-bd44-0b370beffe5c-kube-api-access-l8qc5" pod "0862fa4d-8c1b-4933-bd44-0b370beffe5c" (UID: "0862fa4d-8c1b-4933-bd44-0b370beffe5c"). InnerVolumeSpecName "kube-api-access-l8qc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:19.962227 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:19.962185    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qc5\" (UniqueName: \"kubernetes.io/projected/0862fa4d-8c1b-4933-bd44-0b370beffe5c-kube-api-access-l8qc5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:20.109835 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.109739    5196 scope.go:122] "RemoveContainer" containerID="49c9b71a058711110fbf70f422b90905014f52101e6594176beac87c952d553f"
Nov 03 21:00:20.110085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.110064    5196 scope.go:122] "RemoveContainer" containerID="42ed97163adfe426e920b961ed8d2541f09a7731be2823bee899652ad74cad59"
Nov 03 21:00:20.117750 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.117719    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" path="/var/lib/kubelet/pods/1e2812b4-8772-4e50-a5ff-e09e788fffbe/volumes"
Nov 03 21:00:20.408468 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.408384    5196 generic.go:356] "Generic (PLEG): container finished" podID="704419eb-b6b8-4112-9540-fe79b3991af0" containerID="69148b2ff677f2e33ac0e28934f0e8fbd020a1c62f73835c23248a6490009e1d" exitCode=60
Nov 03 21:00:20.408468 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.408456    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" event={"ID":"704419eb-b6b8-4112-9540-fe79b3991af0","Type":"ContainerDied","Data":"69148b2ff677f2e33ac0e28934f0e8fbd020a1c62f73835c23248a6490009e1d"}
Nov 03 21:00:20.408921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.408519    5196 scope.go:122] "RemoveContainer" containerID="6b57a5b16cb111dba69aca915ada76a89254bab43d1a070a18614a7db1b56520"
Nov 03 21:00:20.409069 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.409050    5196 scope.go:122] "RemoveContainer" containerID="69148b2ff677f2e33ac0e28934f0e8fbd020a1c62f73835c23248a6490009e1d"
Nov 03 21:00:20.409664 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:20.409524    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh_odh-ai-gateway-infra(704419eb-b6b8-4112-9540-fe79b3991af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" podUID="704419eb-b6b8-4112-9540-fe79b3991af0"
Nov 03 21:00:20.411269 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.411239    5196 generic.go:356] "Generic (PLEG): container finished" podID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerID="71780dfefcee52c8050bc466e9b2bd65436506ff41aff4990aa2b10792070b47" exitCode=60
Nov 03 21:00:20.411392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.411313    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" event={"ID":"99a0f4a8-3053-4e62-86ce-68d0cd531ab5","Type":"ContainerDied","Data":"71780dfefcee52c8050bc466e9b2bd65436506ff41aff4990aa2b10792070b47"}
Nov 03 21:00:20.411809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.411790    5196 scope.go:122] "RemoveContainer" containerID="71780dfefcee52c8050bc466e9b2bd65436506ff41aff4990aa2b10792070b47"
Nov 03 21:00:20.412317 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:20.412280    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq_odh-ai-gateway-infra(99a0f4a8-3053-4e62-86ce-68d0cd531ab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5"
Nov 03 21:00:20.417267 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.417226    5196 generic.go:356] "Generic (PLEG): container finished" podID="b5f38d87-0424-422b-859a-97a3636b4004" containerID="0b6d8344c774f1642d76f310046f2b9f1a317a266b699838951ee7de4fae7790" exitCode=60
Nov 03 21:00:20.417408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.417287    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" event={"ID":"b5f38d87-0424-422b-859a-97a3636b4004","Type":"ContainerDied","Data":"0b6d8344c774f1642d76f310046f2b9f1a317a266b699838951ee7de4fae7790"}
Nov 03 21:00:20.417797 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.417774    5196 scope.go:122] "RemoveContainer" containerID="0b6d8344c774f1642d76f310046f2b9f1a317a266b699838951ee7de4fae7790"
Nov 03 21:00:20.418302 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:20.418265    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx_odh-ai-gateway-infra(b5f38d87-0424-422b-859a-97a3636b4004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" podUID="b5f38d87-0424-422b-859a-97a3636b4004"
Nov 03 21:00:20.419027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.419006    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"
Nov 03 21:00:20.419168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.419027    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5" event={"ID":"0862fa4d-8c1b-4933-bd44-0b370beffe5c","Type":"ContainerDied","Data":"769a1fbe10f84de31efb7efc4561a70b8d83eca4ae61ac151fae6654473e028d"}
Nov 03 21:00:20.424892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.424874    5196 scope.go:122] "RemoveContainer" containerID="48f6d7c35d5bb9234a8b04ebfd9b7e4c69d9846ebff4f7395e69121c056ecd63"
Nov 03 21:00:20.443512 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.443490    5196 scope.go:122] "RemoveContainer" containerID="d10b0d076cda365206aee07a39a722b9726d8ec8aad90f69a6f6f34f0dba8bae"
Nov 03 21:00:20.460423 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.460388    5196 scope.go:122] "RemoveContainer" containerID="25618809643f08d5362bd9c251ba8eea45fbae3c6d8e0b76a1d70804fb6c240e"
Nov 03 21:00:20.514786 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.514744    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"]
Nov 03 21:00:20.842944 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.842776    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"]
Nov 03 21:00:20.848390 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.848361    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b119f2-29895660-kwkj5"]
Nov 03 21:00:20.853591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:20.853559    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"]
Nov 03 21:00:21.044559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.044529    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:21.183878 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.183848    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaed4e19-f721-4816-af5e-97fe99adde27-kube-api-access-dxp8d\" (UniqueName: \"kubernetes.io/projected/aaed4e19-f721-4816-af5e-97fe99adde27-kube-api-access-dxp8d\") pod \"aaed4e19-f721-4816-af5e-97fe99adde27\" (UID: \"aaed4e19-f721-4816-af5e-97fe99adde27\") "
Nov 03 21:00:21.184113 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.183912    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaed4e19-f721-4816-af5e-97fe99adde27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed4e19-f721-4816-af5e-97fe99adde27-maas-api-service-ca\") pod \"aaed4e19-f721-4816-af5e-97fe99adde27\" (UID: \"aaed4e19-f721-4816-af5e-97fe99adde27\") "
Nov 03 21:00:21.184885 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.184565    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaed4e19-f721-4816-af5e-97fe99adde27-maas-api-service-ca" pod "aaed4e19-f721-4816-af5e-97fe99adde27" (UID: "aaed4e19-f721-4816-af5e-97fe99adde27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:21.184885 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.184860    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaed4e19-f721-4816-af5e-97fe99adde27-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:21.187147 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.187074    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaed4e19-f721-4816-af5e-97fe99adde27-kube-api-access-dxp8d" pod "aaed4e19-f721-4816-af5e-97fe99adde27" (UID: "aaed4e19-f721-4816-af5e-97fe99adde27"). InnerVolumeSpecName "kube-api-access-dxp8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:21.286902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.286812    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxp8d\" (UniqueName: \"kubernetes.io/projected/aaed4e19-f721-4816-af5e-97fe99adde27-kube-api-access-dxp8d\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:21.434944 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.434911    5196 generic.go:356] "Generic (PLEG): container finished" podID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerID="c2f680ebdcef82e57be2dc90b4f9e670439f5b3898182b85d64a89eaf6371f1b" exitCode=60
Nov 03 21:00:21.435395 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.434996    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" event={"ID":"c95f6b0e-1edb-47c8-9ed2-201f135db605","Type":"ContainerDied","Data":"c2f680ebdcef82e57be2dc90b4f9e670439f5b3898182b85d64a89eaf6371f1b"}
Nov 03 21:00:21.435395 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.435160    5196 scope.go:122] "RemoveContainer" containerID="49c9b71a058711110fbf70f422b90905014f52101e6594176beac87c952d553f"
Nov 03 21:00:21.435676 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.435654    5196 scope.go:122] "RemoveContainer" containerID="c2f680ebdcef82e57be2dc90b4f9e670439f5b3898182b85d64a89eaf6371f1b"
Nov 03 21:00:21.436203 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:21.436166    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69_odh-ai-gateway-infra(c95f6b0e-1edb-47c8-9ed2-201f135db605)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605"
Nov 03 21:00:21.436911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.436891    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"
Nov 03 21:00:21.436999 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.436903    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g" event={"ID":"aaed4e19-f721-4816-af5e-97fe99adde27","Type":"ContainerDied","Data":"8154578596ac280a47342bf2290c5b0d8d4fd4decf0493e374761b0ecaba6e69"}
Nov 03 21:00:21.439281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.439259    5196 generic.go:356] "Generic (PLEG): container finished" podID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerID="a18c6d8943fea9b272ebe81658975a60f7a44918d4e5814881393aa3d6e1f5eb" exitCode=60
Nov 03 21:00:21.439391 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.439335    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" event={"ID":"b7c4aead-2076-4cbf-9322-8d472ccf258d","Type":"ContainerDied","Data":"a18c6d8943fea9b272ebe81658975a60f7a44918d4e5814881393aa3d6e1f5eb"}
Nov 03 21:00:21.439852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.439831    5196 scope.go:122] "RemoveContainer" containerID="a18c6d8943fea9b272ebe81658975a60f7a44918d4e5814881393aa3d6e1f5eb"
Nov 03 21:00:21.440327 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:21.440302    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf_odh-ai-gateway-infra(b7c4aead-2076-4cbf-9322-8d472ccf258d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d"
Nov 03 21:00:21.510963 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.508347    5196 scope.go:122] "RemoveContainer" containerID="b940f5012f2218adfeed103fac96c5ff5d78130eb7e2771e42910efa12a1bd84"
Nov 03 21:00:21.652860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.634454    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"]
Nov 03 21:00:21.738702 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.734922    5196 scope.go:122] "RemoveContainer" containerID="42ed97163adfe426e920b961ed8d2541f09a7731be2823bee899652ad74cad59"
Nov 03 21:00:21.760715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.759238    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"]
Nov 03 21:00:21.853559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.853528    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:21.973043 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.973007    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:21.992108 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:21.992050    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"]
Nov 03 21:00:22.000883 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.000851    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b6d800-29895660-5f64g"]
Nov 03 21:00:22.007585 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.007559    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:22.016135 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016062    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f38d87-0424-422b-859a-97a3636b4004-kube-api-access-wszxc\" (UniqueName: \"kubernetes.io/projected/b5f38d87-0424-422b-859a-97a3636b4004-kube-api-access-wszxc\") pod \"b5f38d87-0424-422b-859a-97a3636b4004\" (UID: \"b5f38d87-0424-422b-859a-97a3636b4004\") "
Nov 03 21:00:22.016288 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016141    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-maas-api-service-ca\") pod \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\" (UID: \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\") "
Nov 03 21:00:22.016288 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016193    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704419eb-b6b8-4112-9540-fe79b3991af0-kube-api-access-88knz\" (UniqueName: \"kubernetes.io/projected/704419eb-b6b8-4112-9540-fe79b3991af0-kube-api-access-88knz\") pod \"704419eb-b6b8-4112-9540-fe79b3991af0\" (UID: \"704419eb-b6b8-4112-9540-fe79b3991af0\") "
Nov 03 21:00:22.016288 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016232    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704419eb-b6b8-4112-9540-fe79b3991af0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704419eb-b6b8-4112-9540-fe79b3991af0-maas-api-service-ca\") pod \"704419eb-b6b8-4112-9540-fe79b3991af0\" (UID: \"704419eb-b6b8-4112-9540-fe79b3991af0\") "
Nov 03 21:00:22.016487 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016435    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-kube-api-access-wlrbs\" (UniqueName: \"kubernetes.io/projected/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-kube-api-access-wlrbs\") pod \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\" (UID: \"99a0f4a8-3053-4e62-86ce-68d0cd531ab5\") "
Nov 03 21:00:22.016549 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016495    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f38d87-0424-422b-859a-97a3636b4004-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f38d87-0424-422b-859a-97a3636b4004-maas-api-service-ca\") pod \"b5f38d87-0424-422b-859a-97a3636b4004\" (UID: \"b5f38d87-0424-422b-859a-97a3636b4004\") "
Nov 03 21:00:22.016603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016558    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-maas-api-service-ca" pod "99a0f4a8-3053-4e62-86ce-68d0cd531ab5" (UID: "99a0f4a8-3053-4e62-86ce-68d0cd531ab5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:22.016701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016648    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704419eb-b6b8-4112-9540-fe79b3991af0-maas-api-service-ca" pod "704419eb-b6b8-4112-9540-fe79b3991af0" (UID: "704419eb-b6b8-4112-9540-fe79b3991af0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:22.016891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.016867    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f38d87-0424-422b-859a-97a3636b4004-maas-api-service-ca" pod "b5f38d87-0424-422b-859a-97a3636b4004" (UID: "b5f38d87-0424-422b-859a-97a3636b4004"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:22.017476 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.017453    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f38d87-0424-422b-859a-97a3636b4004-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:22.017571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.017478    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:22.017571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.017492    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704419eb-b6b8-4112-9540-fe79b3991af0-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:22.018637 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.018616    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704419eb-b6b8-4112-9540-fe79b3991af0-kube-api-access-88knz" pod "704419eb-b6b8-4112-9540-fe79b3991af0" (UID: "704419eb-b6b8-4112-9540-fe79b3991af0"). InnerVolumeSpecName "kube-api-access-88knz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:22.018715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.018651    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f38d87-0424-422b-859a-97a3636b4004-kube-api-access-wszxc" pod "b5f38d87-0424-422b-859a-97a3636b4004" (UID: "b5f38d87-0424-422b-859a-97a3636b4004"). InnerVolumeSpecName "kube-api-access-wszxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:22.019131 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.019069    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-kube-api-access-wlrbs" pod "99a0f4a8-3053-4e62-86ce-68d0cd531ab5" (UID: "99a0f4a8-3053-4e62-86ce-68d0cd531ab5"). InnerVolumeSpecName "kube-api-access-wlrbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:22.117446 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.117405    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" path="/var/lib/kubelet/pods/0862fa4d-8c1b-4933-bd44-0b370beffe5c/volumes"
Nov 03 21:00:22.118287 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.118265    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" path="/var/lib/kubelet/pods/aaed4e19-f721-4816-af5e-97fe99adde27/volumes"
Nov 03 21:00:22.119038 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.119017    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wszxc\" (UniqueName: \"kubernetes.io/projected/b5f38d87-0424-422b-859a-97a3636b4004-kube-api-access-wszxc\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:22.119210 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.119194    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88knz\" (UniqueName: \"kubernetes.io/projected/704419eb-b6b8-4112-9540-fe79b3991af0-kube-api-access-88knz\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:22.119279 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.119217    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlrbs\" (UniqueName: \"kubernetes.io/projected/99a0f4a8-3053-4e62-86ce-68d0cd531ab5-kube-api-access-wlrbs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:22.458895 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.458864    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"
Nov 03 21:00:22.459562 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.458923    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh" event={"ID":"704419eb-b6b8-4112-9540-fe79b3991af0","Type":"ContainerDied","Data":"202ae1131e3188bf8430fa73e0b3158747ce73ee8fe0f90d0b11b82582e4a868"}
Nov 03 21:00:22.459562 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.458982    5196 scope.go:122] "RemoveContainer" containerID="69148b2ff677f2e33ac0e28934f0e8fbd020a1c62f73835c23248a6490009e1d"
Nov 03 21:00:22.461538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.461511    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"
Nov 03 21:00:22.461672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.461535    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq" event={"ID":"99a0f4a8-3053-4e62-86ce-68d0cd531ab5","Type":"ContainerDied","Data":"e24c9bc34dfc199d690d2968be1d25c77b8d20ff0aeab1aa1b05979250277876"}
Nov 03 21:00:22.472404 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.472367    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx" event={"ID":"b5f38d87-0424-422b-859a-97a3636b4004","Type":"ContainerDied","Data":"290d61883965cd8319f0fc6fc3331469ec9e999ba10adaf788cb34e1219b8ae9"}
Nov 03 21:00:22.472590 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.472490    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"
Nov 03 21:00:22.475226 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.475178    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:22.479897 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.479877    5196 scope.go:122] "RemoveContainer" containerID="71780dfefcee52c8050bc466e9b2bd65436506ff41aff4990aa2b10792070b47"
Nov 03 21:00:22.493018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.492998    5196 scope.go:122] "RemoveContainer" containerID="0b6d8344c774f1642d76f310046f2b9f1a317a266b699838951ee7de4fae7790"
Nov 03 21:00:22.622178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.621998    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"]
Nov 03 21:00:22.627070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.627040    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fe2825-29895660-7dxbh"]
Nov 03 21:00:22.725387 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.725294    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"]
Nov 03 21:00:22.730307 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.730273    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-211d68bf-29895660-sdznx"]
Nov 03 21:00:22.779334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.779296    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"]
Nov 03 21:00:22.832537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.832503    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"]
Nov 03 21:00:22.840118 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.840065    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-41aa21-29895660-mv8tq"]
Nov 03 21:00:22.896021 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:22.895986    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"]
Nov 03 21:00:23.006661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.006331    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:23.138837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.138804    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95f6b0e-1edb-47c8-9ed2-201f135db605-kube-api-access-gwk86\" (UniqueName: \"kubernetes.io/projected/c95f6b0e-1edb-47c8-9ed2-201f135db605-kube-api-access-gwk86\") pod \"c95f6b0e-1edb-47c8-9ed2-201f135db605\" (UID: \"c95f6b0e-1edb-47c8-9ed2-201f135db605\") "
Nov 03 21:00:23.139105 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.139077    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95f6b0e-1edb-47c8-9ed2-201f135db605-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95f6b0e-1edb-47c8-9ed2-201f135db605-maas-api-service-ca\") pod \"c95f6b0e-1edb-47c8-9ed2-201f135db605\" (UID: \"c95f6b0e-1edb-47c8-9ed2-201f135db605\") "
Nov 03 21:00:23.139496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.139464    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95f6b0e-1edb-47c8-9ed2-201f135db605-maas-api-service-ca" pod "c95f6b0e-1edb-47c8-9ed2-201f135db605" (UID: "c95f6b0e-1edb-47c8-9ed2-201f135db605"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:23.139672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.139653    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c95f6b0e-1edb-47c8-9ed2-201f135db605-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:23.141273 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.141246    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95f6b0e-1edb-47c8-9ed2-201f135db605-kube-api-access-gwk86" pod "c95f6b0e-1edb-47c8-9ed2-201f135db605" (UID: "c95f6b0e-1edb-47c8-9ed2-201f135db605"). InnerVolumeSpecName "kube-api-access-gwk86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:23.154536 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.154508    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:23.241345 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.241247    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c4aead-2076-4cbf-9322-8d472ccf258d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4aead-2076-4cbf-9322-8d472ccf258d-maas-api-service-ca\") pod \"b7c4aead-2076-4cbf-9322-8d472ccf258d\" (UID: \"b7c4aead-2076-4cbf-9322-8d472ccf258d\") "
Nov 03 21:00:23.241570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.241552    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c4aead-2076-4cbf-9322-8d472ccf258d-kube-api-access-zvr2l\" (UniqueName: \"kubernetes.io/projected/b7c4aead-2076-4cbf-9322-8d472ccf258d-kube-api-access-zvr2l\") pod \"b7c4aead-2076-4cbf-9322-8d472ccf258d\" (UID: \"b7c4aead-2076-4cbf-9322-8d472ccf258d\") "
Nov 03 21:00:23.241731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.241709    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c4aead-2076-4cbf-9322-8d472ccf258d-maas-api-service-ca" pod "b7c4aead-2076-4cbf-9322-8d472ccf258d" (UID: "b7c4aead-2076-4cbf-9322-8d472ccf258d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:23.242743 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.242719    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4aead-2076-4cbf-9322-8d472ccf258d-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:23.242914 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.242894    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwk86\" (UniqueName: \"kubernetes.io/projected/c95f6b0e-1edb-47c8-9ed2-201f135db605-kube-api-access-gwk86\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:23.245085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.245052    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c4aead-2076-4cbf-9322-8d472ccf258d-kube-api-access-zvr2l" pod "b7c4aead-2076-4cbf-9322-8d472ccf258d" (UID: "b7c4aead-2076-4cbf-9322-8d472ccf258d"). InnerVolumeSpecName "kube-api-access-zvr2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:23.345011 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.344977    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvr2l\" (UniqueName: \"kubernetes.io/projected/b7c4aead-2076-4cbf-9322-8d472ccf258d-kube-api-access-zvr2l\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:23.485491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.485461    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"
Nov 03 21:00:23.485960 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.485500    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69" event={"ID":"c95f6b0e-1edb-47c8-9ed2-201f135db605","Type":"ContainerDied","Data":"6fe522b572d12725a114f03773c1715dd29cef2a1528096ab62bf12c2b9c811f"}
Nov 03 21:00:23.485960 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.485577    5196 scope.go:122] "RemoveContainer" containerID="c2f680ebdcef82e57be2dc90b4f9e670439f5b3898182b85d64a89eaf6371f1b"
Nov 03 21:00:23.487202 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.487167    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf" event={"ID":"b7c4aead-2076-4cbf-9322-8d472ccf258d","Type":"ContainerDied","Data":"24b460cbbff3073f75bcbe8e2987de351aa20db76184e77dc76be1381813aa45"}
Nov 03 21:00:23.487352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.487185    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"
Nov 03 21:00:23.498312 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.498149    5196 scope.go:122] "RemoveContainer" containerID="a18c6d8943fea9b272ebe81658975a60f7a44918d4e5814881393aa3d6e1f5eb"
Nov 03 21:00:23.643006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.642972    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"]
Nov 03 21:00:23.647754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.647726    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1ec892-29895660-r8z69"]
Nov 03 21:00:23.748045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.747960    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"]
Nov 03 21:00:23.753888 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:23.753857    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1ec1b6-29895660-c8bcf"]
Nov 03 21:00:24.112566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:24.112527    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" path="/var/lib/kubelet/pods/704419eb-b6b8-4112-9540-fe79b3991af0/volumes"
Nov 03 21:00:24.113177 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:24.113155    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" path="/var/lib/kubelet/pods/99a0f4a8-3053-4e62-86ce-68d0cd531ab5/volumes"
Nov 03 21:00:24.113668 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:24.113649    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f38d87-0424-422b-859a-97a3636b4004" path="/var/lib/kubelet/pods/b5f38d87-0424-422b-859a-97a3636b4004/volumes"
Nov 03 21:00:24.114502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:24.114480    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" path="/var/lib/kubelet/pods/b7c4aead-2076-4cbf-9322-8d472ccf258d/volumes"
Nov 03 21:00:24.114952 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:24.114933    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" path="/var/lib/kubelet/pods/c95f6b0e-1edb-47c8-9ed2-201f135db605/volumes"
Nov 03 21:00:25.749453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:25.749414    5196 scope.go:122] "RemoveContainer" containerID="d55a9945d73ba02909af75cdb288e0420244b45397ab98a861c407063654825a"
Nov 03 21:00:27.599259 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:27.599222    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:29.548646 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:29.548607    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:00:32.412079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:32.412032    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:37.289029 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:37.288997    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:37.736816 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:37.736782    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:38.175253 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:38.175208    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:38.773376 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:38.773331    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:39.413642 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:39.412865    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:40.564173 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:40.564136    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:42.306748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:42.306706    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:42.634854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:42.634817    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:42.987236 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:42.978641    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:43.435232 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:43.435197    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:46.373614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:46.373575    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:47.187074 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:47.187040    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:47.407449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:47.407410    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:47.557681 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:47.557588    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:47.884736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:47.884697    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:48.035248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:48.035209    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:48.354681 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:48.354641    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:48.487188 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:00:48.487146    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:00:49.012118 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:49.002811    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:49.705260 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:49.705210    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:50.494293 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:50.494245    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:50.725791 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:50.725733    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:52.329634 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:52.329593    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:52.752840 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:52.752737    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:52.937407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:52.937367    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:53.102721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:53.102678    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:53.481437 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:53.481344    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:53.644404 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:53.644366    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:55.617558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:55.617469    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:55.837677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:55.837624    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:56.000806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:56.000708    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:56.375216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:56.375179    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:57.072413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:57.072372    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:57.684408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:57.684370    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:57.887551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:57.887513    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:58.850333 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:58.850299    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"]
Nov 03 21:00:58.850890 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:58.850509    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy" containerID="cri-o://1af1a0f3ada2b5eafdc3b5f2b67178f3634ef4999b64d8fc1d7c6cab583aec9e" gracePeriod=30
Nov 03 21:00:58.881182 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:58.881144    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:59.197571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:59.197484    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:59.380768 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:59.380723    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:59.579646 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:00:59.579614    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:00.241167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:00.241073    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:00.504700 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:00.504449    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:03.995805 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:03.995776    5196 generic.go:356] "Generic (PLEG): container finished" podID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerID="1af1a0f3ada2b5eafdc3b5f2b67178f3634ef4999b64d8fc1d7c6cab583aec9e" exitCode=0
Nov 03 21:01:03.996300 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:03.995851    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" event={"ID":"5c6f4c3f-8932-4cdf-a1b5-fd829035d788","Type":"ContainerDied","Data":"1af1a0f3ada2b5eafdc3b5f2b67178f3634ef4999b64d8fc1d7c6cab583aec9e"}
Nov 03 21:01:04.167787 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.167760    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 21:01:04.204406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.204369    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istiod-ca-cert\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.204596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.204439    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-envoy\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.204641 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.204607    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-podinfo\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.204736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.204719    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-certs\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.204779 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.204760    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-token\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.204827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.204764    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istiod-ca-cert" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:04.204953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.204935    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-socket\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.205030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.204973    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-certs" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:04.205087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.205072    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-kube-api-access-fj95z\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-kube-api-access-fj95z\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.205211 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.205160    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-credential-socket\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.205211 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.205204    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-data\") pod \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\" (UID: \"5c6f4c3f-8932-4cdf-a1b5-fd829035d788\") "
Nov 03 21:01:04.205489 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.205216    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-socket" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:04.205489 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.205471    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-credential-socket" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:04.205739 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.205718    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-data" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:04.206321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.206162    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.206321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.206187    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.206321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.206204    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.206321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.206217    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.206321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.206232    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.207411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.207387    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-envoy" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:04.207532 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.207422    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-token" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:04.207600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.207577    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-podinfo" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:04.207867 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.207848    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-kube-api-access-fj95z" pod "5c6f4c3f-8932-4cdf-a1b5-fd829035d788" (UID: "5c6f4c3f-8932-4cdf-a1b5-fd829035d788"). InnerVolumeSpecName "kube-api-access-fj95z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:04.308633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.308596    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.308633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.308629    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj95z\" (UniqueName: \"kubernetes.io/projected/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-kube-api-access-fj95z\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.308633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.308641    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.308957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.308650    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c6f4c3f-8932-4cdf-a1b5-fd829035d788-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:04.363823 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:04.363781    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:05.006198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:05.006167    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"
Nov 03 21:01:05.006641 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:05.006164    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v" event={"ID":"5c6f4c3f-8932-4cdf-a1b5-fd829035d788","Type":"ContainerDied","Data":"a77dbca93d14eae8d91d0bedfc6207da6a3542b52c2c61ee67720c72b836809c"}
Nov 03 21:01:05.006641 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:05.006313    5196 scope.go:122] "RemoveContainer" containerID="1af1a0f3ada2b5eafdc3b5f2b67178f3634ef4999b64d8fc1d7c6cab583aec9e"
Nov 03 21:01:05.164912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:05.164878    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"]
Nov 03 21:01:05.211838 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:05.211806    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-hfd9v"]
Nov 03 21:01:06.111827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:06.111798    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" path="/var/lib/kubelet/pods/5c6f4c3f-8932-4cdf-a1b5-fd829035d788/volumes"
Nov 03 21:01:09.289118 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:09.289068    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:09.646502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:09.646465    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:09.884590 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:09.867136    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:13.961930 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:13.961889    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"]
Nov 03 21:01:13.962433 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:13.962230    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy" containerID="cri-o://b613846a3458598d2d3d5d78b863e326656319646c823619d888fc34456ed227" gracePeriod=30
Nov 03 21:01:14.075854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:14.075810    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:14.235184 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:14.231634    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:14.380875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:14.380843    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:16.153381 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:16.153348    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"]
Nov 03 21:01:16.153834 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:16.153566    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy" containerID="cri-o://64d87e957d50f1882043d3a5f426722366ee2feead604bf78eeed38ad827c214" gracePeriod=30
Nov 03 21:01:18.487339 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:01:18.487296    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:01:18.946016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:18.945962    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:19.256696 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.256663    5196 generic.go:356] "Generic (PLEG): container finished" podID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerID="b613846a3458598d2d3d5d78b863e326656319646c823619d888fc34456ed227" exitCode=0
Nov 03 21:01:19.256854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.256732    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" event={"ID":"cb74f130-9b42-4f99-b7ee-59a64b93d4da","Type":"ContainerDied","Data":"b613846a3458598d2d3d5d78b863e326656319646c823619d888fc34456ed227"}
Nov 03 21:01:19.256854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.256784    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v" event={"ID":"cb74f130-9b42-4f99-b7ee-59a64b93d4da","Type":"ContainerDied","Data":"90476724bcab730946f0ed72f892c1770cc31f9a55313649f9026ab303f185b5"}
Nov 03 21:01:19.256854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.256801    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90476724bcab730946f0ed72f892c1770cc31f9a55313649f9026ab303f185b5"
Nov 03 21:01:19.287271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.287243    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 21:01:19.379385 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379345    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-envoy\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.379594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379417    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-kube-api-access-bb7hn\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-kube-api-access-bb7hn\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.379594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379449    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-certs\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.379594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379502    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-token\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-token\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.379801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379671    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-data\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.379801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379739    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-socket\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.379896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379842    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istiod-ca-cert\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.379957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379908    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-podinfo\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.380020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.379994    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-certs" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:19.380082 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380052    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-credential-socket\") pod \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\" (UID: \"cb74f130-9b42-4f99-b7ee-59a64b93d4da\") "
Nov 03 21:01:19.380249 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380226    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-socket" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:19.380433 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380413    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-data" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:19.380520 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380446    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istiod-ca-cert" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:19.380612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380593    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-credential-socket" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:19.380849 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380828    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.380941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380853    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.380941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380864    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.380941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380879    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.380941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.380897    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.388141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.383075    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-kube-api-access-bb7hn" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "kube-api-access-bb7hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:19.388141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.383144    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-envoy" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:19.388141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.383647    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-token" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:19.388141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.384503    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-podinfo" pod "cb74f130-9b42-4f99-b7ee-59a64b93d4da" (UID: "cb74f130-9b42-4f99-b7ee-59a64b93d4da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:19.482164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.482087    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.482164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.482163    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.482382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.482175    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb7hn\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-kube-api-access-bb7hn\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.482382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:19.482185    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb74f130-9b42-4f99-b7ee-59a64b93d4da-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:20.266074 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:20.266038    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"
Nov 03 21:01:20.473351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:20.473267    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"]
Nov 03 21:01:20.519740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:20.519704    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-txz9v"]
Nov 03 21:01:21.278446 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.278418    5196 generic.go:356] "Generic (PLEG): container finished" podID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerID="64d87e957d50f1882043d3a5f426722366ee2feead604bf78eeed38ad827c214" exitCode=0
Nov 03 21:01:21.278787 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.278493    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" event={"ID":"3eb104ce-df5c-489d-b5d8-2d6a41b8c218","Type":"ContainerDied","Data":"64d87e957d50f1882043d3a5f426722366ee2feead604bf78eeed38ad827c214"}
Nov 03 21:01:21.459220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.459194    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 21:01:21.523239 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523186    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-kube-api-access-zrvmg\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-kube-api-access-zrvmg\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.523426 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523290    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istiod-ca-cert\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.523475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523441    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-podinfo\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.523525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523490    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-token\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-token\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.523583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523549    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-credential-socket\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.523647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523634    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-socket\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.523736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523694    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istiod-ca-cert" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:21.523913 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523887    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-credential-socket" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.524016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.523981    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-socket" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.524352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.524286    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.524352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.524315    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.524352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.524330    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.525990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.525969    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-podinfo" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:21.526142 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.525999    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-kube-api-access-zrvmg" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "kube-api-access-zrvmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:21.526238 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.526221    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-token" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:21.624961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.624922    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-envoy\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.625291 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.624980    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-data\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.625291 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.625021    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-certs\") pod \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\" (UID: \"3eb104ce-df5c-489d-b5d8-2d6a41b8c218\") "
Nov 03 21:01:21.625886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.625856    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvmg\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-kube-api-access-zrvmg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.626006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.625891    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.626006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.625906    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.626222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.626196    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-certs" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.626222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.626204    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-data" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.628040 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.628014    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-envoy" pod "3eb104ce-df5c-489d-b5d8-2d6a41b8c218" (UID: "3eb104ce-df5c-489d-b5d8-2d6a41b8c218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.729558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.729466    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.729558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.729495    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.729558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:21.729504    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb104ce-df5c-489d-b5d8-2d6a41b8c218-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.111320 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:22.111280    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" path="/var/lib/kubelet/pods/cb74f130-9b42-4f99-b7ee-59a64b93d4da/volumes"
Nov 03 21:01:22.289510 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:22.289471    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"
Nov 03 21:01:22.289999 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:22.289479    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk" event={"ID":"3eb104ce-df5c-489d-b5d8-2d6a41b8c218","Type":"ContainerDied","Data":"9c735078b3585cd939247d075bef7abcede114b11ca3de2678ba9214209e77e6"}
Nov 03 21:01:22.289999 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:22.289614    5196 scope.go:122] "RemoveContainer" containerID="64d87e957d50f1882043d3a5f426722366ee2feead604bf78eeed38ad827c214"
Nov 03 21:01:22.495129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:22.495005    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"]
Nov 03 21:01:22.542277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:22.542244    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-54fjk"]
Nov 03 21:01:23.550052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:23.550019    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:23.775957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:23.775465    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:23.939798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:23.939763    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:24.111194 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:24.111165    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" path="/var/lib/kubelet/pods/3eb104ce-df5c-489d-b5d8-2d6a41b8c218/volumes"
Nov 03 21:01:25.843905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:25.843875    5196 scope.go:122] "RemoveContainer" containerID="a0191b3e9efd97958290389b47343c539eed27625523df5613d9e1977cde99cf"
Nov 03 21:01:25.856755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:25.856734    5196 scope.go:122] "RemoveContainer" containerID="81233f0733369ad39a81b026107804379358f564f6642c0b0cc441c9cc4028bb"
Nov 03 21:01:25.871421 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:25.871396    5196 scope.go:122] "RemoveContainer" containerID="fb8ed6e0d4ac84b195e318a828680ecba7ac2e2c7bda4fe090af98dd1fe77fbe"
Nov 03 21:01:25.882754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:25.882732    5196 scope.go:122] "RemoveContainer" containerID="b613846a3458598d2d3d5d78b863e326656319646c823619d888fc34456ed227"
Nov 03 21:01:25.894342 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:25.894315    5196 scope.go:122] "RemoveContainer" containerID="95cea79ab72c62a50537985a62a95af90539539e8ab82744a97a73a9eb733336"
Nov 03 21:01:28.472049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:28.472003    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:33.133443 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:33.133400    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:01:33.278302 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:33.278268    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:38.013118 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:38.012185    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:42.726142 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:42.725695    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:46.701495 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:46.701453    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:47.488311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:47.488268    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:48.487536 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:01:48.487496    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:01:52.272564 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:52.272532    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:56.972337 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:01:56.972302    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:01.699786 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:01.699744    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:01.884278 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:01.884159    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:02.030527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:02.030438    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:06.582904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:06.582866    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:11.251236 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:11.251197    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:11.361438 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:11.361300    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:02:11.406134 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:11.406086    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:11.723612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:11.723578    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:16.115269 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:16.115233    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:16.314459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:16.314421    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:16.470642 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:16.470550    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:18.491388 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:02:18.491339    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:02:21.164260 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:21.164219    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:25.690496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:25.690412    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:25.896193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:25.896147    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:25.964933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:25.964840    5196 scope.go:122] "RemoveContainer" containerID="8c89741cd40489a7a8e0132b2b4b76aab2bd489cd8501af28d3531bea98e21ad"
Nov 03 21:02:26.044783 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:26.044741    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:30.646327 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:30.646293    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:35.377418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:35.377378    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:40.297121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:40.296342    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:45.049573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:45.049525    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:47.021950 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:47.021900    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:47.272953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:47.272859    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:48.487661 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:02:48.487625    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:02:49.877546 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:49.877506    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:53.996176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:53.996138    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:54.187907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:54.187865    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:55.002929 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:55.002887    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:55.433600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:55.433562    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:56.129155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:56.129117    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:58.009056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:58.009020    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:58.207460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:02:58.207377    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:00.502860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.502817    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:00.937314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.937275    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7280-a3fe47"]
Nov 03 21:03:00.938257 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938230    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy"
Nov 03 21:03:00.938257 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938254    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938268    5196 cpu_manager.go:414] "containerMap: removing container" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938276    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938287    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938294    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938306    5196 cpu_manager.go:414] "containerMap: removing container" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938313    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938321    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938330    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938340    5196 cpu_manager.go:414] "containerMap: removing container" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938347    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938359    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938366    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938374    5196 cpu_manager.go:414] "containerMap: removing container" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938381    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938399    5196 cpu_manager.go:414] "containerMap: removing container" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938405    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938425    5196 cpu_manager.go:414] "containerMap: removing container" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938434    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938448    5196 cpu_manager.go:414] "containerMap: removing container" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938455    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938465    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938471    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.938469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938480    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938487    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938504    5196 cpu_manager.go:414] "containerMap: removing container" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938512    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938523    5196 cpu_manager.go:414] "containerMap: removing container" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938529    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938547    5196 cpu_manager.go:414] "containerMap: removing container" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938553    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938567    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938574    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938594    5196 cpu_manager.go:414] "containerMap: removing container" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938600    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938618    5196 cpu_manager.go:414] "containerMap: removing container" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938625    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938643    5196 cpu_manager.go:414] "containerMap: removing container" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938650    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938674    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938681    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938695    5196 cpu_manager.go:414] "containerMap: removing container" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938701    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938712    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938719    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938735    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938743    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938933    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938945    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938956    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938964    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938974    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6f4c3f-8932-4cdf-a1b5-fd829035d788" containerName="istio-proxy"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938986    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.938994    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939002    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939009    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939018    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939026    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939036    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb74f130-9b42-4f99-b7ee-59a64b93d4da" containerName="istio-proxy"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939046    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939056    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939062    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c4aead-2076-4cbf-9322-8d472ccf258d" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939070    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f38d87-0424-422b-859a-97a3636b4004" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939079    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939109    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="704419eb-b6b8-4112-9540-fe79b3991af0" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939122    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaed4e19-f721-4816-af5e-97fe99adde27" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939133    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939145    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939156    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939167    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939176    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:00.939654 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.939186    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb104ce-df5c-489d-b5d8-2d6a41b8c218" containerName="istio-proxy"
Nov 03 21:03:00.943843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.943820    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-a3fe47"
Nov 03 21:03:00.976724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.976676    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pc7h\" (UniqueName: \"kubernetes.io/projected/2c0e02d6-e9e6-471c-af19-8d1006d6d813-kube-api-access-4pc7h\") pod \"test-curl-7280-a3fe47\" (UID: \"2c0e02d6-e9e6-471c-af19-8d1006d6d813\") " pod="opendatahub/test-curl-7280-a3fe47"
Nov 03 21:03:00.992767 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:00.992734    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7280-a3fe47"]
Nov 03 21:03:01.034296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:01.034257    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-24vlg\""
Nov 03 21:03:01.078829 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:01.078790    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pc7h\" (UniqueName: \"kubernetes.io/projected/2c0e02d6-e9e6-471c-af19-8d1006d6d813-kube-api-access-4pc7h\") pod \"test-curl-7280-a3fe47\" (UID: \"2c0e02d6-e9e6-471c-af19-8d1006d6d813\") " pod="opendatahub/test-curl-7280-a3fe47"
Nov 03 21:03:01.132490 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:01.132457    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pc7h\" (UniqueName: \"kubernetes.io/projected/2c0e02d6-e9e6-471c-af19-8d1006d6d813-kube-api-access-4pc7h\") pod \"test-curl-7280-a3fe47\" (UID: \"2c0e02d6-e9e6-471c-af19-8d1006d6d813\") " pod="opendatahub/test-curl-7280-a3fe47"
Nov 03 21:03:01.259568 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:01.259479    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-a3fe47"
Nov 03 21:03:01.643956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:01.643799    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7280-a3fe47"]
Nov 03 21:03:01.647693 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:03:01.647663    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c0e02d6_e9e6_471c_af19_8d1006d6d813.slice/crio-bba164988ab9949f42e3c559572f6d7963856cf5f952a0fbb20c2f2f68fcf8bc WatchSource:0}: Error finding container bba164988ab9949f42e3c559572f6d7963856cf5f952a0fbb20c2f2f68fcf8bc: Status 404 returned error can't find the container with id bba164988ab9949f42e3c559572f6d7963856cf5f952a0fbb20c2f2f68fcf8bc
Nov 03 21:03:02.408411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:02.408369    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7280-a3fe47" event={"ID":"2c0e02d6-e9e6-471c-af19-8d1006d6d813","Type":"ContainerStarted","Data":"1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf"}
Nov 03 21:03:02.408620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:02.408423    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7280-a3fe47" event={"ID":"2c0e02d6-e9e6-471c-af19-8d1006d6d813","Type":"ContainerStarted","Data":"bba164988ab9949f42e3c559572f6d7963856cf5f952a0fbb20c2f2f68fcf8bc"}
Nov 03 21:03:02.441992 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:02.441954    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:02.518762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:02.518703    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7280-a3fe47" podStartSLOduration=2.518688289 podStartE2EDuration="2.518688289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:02.514394638 +0000 UTC m=+2022.518353386" watchObservedRunningTime="2026-11-03 21:03:02.518688289 +0000 UTC m=+2022.522647013"
Nov 03 21:03:02.657854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:02.657814    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:02.821352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:02.821317    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:03.913597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:03.913563    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7280-a3fe47"]
Nov 03 21:03:03.918261 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:03.918235    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7280-a3fe47"]
Nov 03 21:03:04.299967 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:04.299852    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:04.426230 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:04.426195    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7280-a3fe47" podUID="2c0e02d6-e9e6-471c-af19-8d1006d6d813" containerName="test-curl-7280-a3fe47" containerID="cri-o://1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf" gracePeriod=2
Nov 03 21:03:05.041278 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:05.041224    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:05.673629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:05.673584    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:05.848618 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:05.848582    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:06.039614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.039481    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:06.051190 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.051158    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"]
Nov 03 21:03:06.052655 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052626    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:06.052655 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052652    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:06.052871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052692    5196 cpu_manager.go:414] "containerMap: removing container" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:06.052871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052700    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2812b4-8772-4e50-a5ff-e09e788fffbe" containerName="cleanup"
Nov 03 21:03:06.052871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052712    5196 cpu_manager.go:414] "containerMap: removing container" podUID="2c0e02d6-e9e6-471c-af19-8d1006d6d813" containerName="test-curl-7280-a3fe47"
Nov 03 21:03:06.052871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052720    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0e02d6-e9e6-471c-af19-8d1006d6d813" containerName="test-curl-7280-a3fe47"
Nov 03 21:03:06.052871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052771    5196 cpu_manager.go:414] "containerMap: removing container" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:06.052871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052778    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0862fa4d-8c1b-4933-bd44-0b370beffe5c" containerName="cleanup"
Nov 03 21:03:06.053197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.052986    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0e02d6-e9e6-471c-af19-8d1006d6d813" containerName="test-curl-7280-a3fe47"
Nov 03 21:03:06.053197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.053007    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95f6b0e-1edb-47c8-9ed2-201f135db605" containerName="cleanup"
Nov 03 21:03:06.053197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.053026    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a0f4a8-3053-4e62-86ce-68d0cd531ab5" containerName="cleanup"
Nov 03 21:03:06.057074 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.057051    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.176550 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.176466    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:03:06.177449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.177332    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:06.198128 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.197934    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:06.203451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.203417    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"]
Nov 03 21:03:06.232532 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.225670    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gcx\" (UniqueName: \"kubernetes.io/projected/8a255c4d-6f63-4166-afe3-7598df773d34-kube-api-access-c8gcx\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.236755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.236678    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.236965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.236935    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.342129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.340190    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.342129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.340422    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.342129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.340587    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gcx\" (UniqueName: \"kubernetes.io/projected/8a255c4d-6f63-4166-afe3-7598df773d34-kube-api-access-c8gcx\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.342129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.341794    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.387309 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.386590    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.404008 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.403970    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gcx\" (UniqueName: \"kubernetes.io/projected/8a255c4d-6f63-4166-afe3-7598df773d34-kube-api-access-c8gcx\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.406850 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.406344    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"]
Nov 03 21:03:06.413224 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.413200    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:06.461974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.461543    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"]
Nov 03 21:03:06.462404 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.462376    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c571ef\""
Nov 03 21:03:06.462606 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.462586    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c571ef-dockercfg-92dbn\""
Nov 03 21:03:06.565573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.565521    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffeef1b-d187-4e98-a791-d47cc5155c8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr\" (UID: \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:06.565793 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.565693    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2cq4\" (UniqueName: \"kubernetes.io/projected/5ffeef1b-d187-4e98-a791-d47cc5155c8e-kube-api-access-q2cq4\") pod \"payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr\" (UID: \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:06.585071 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.583662    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"]
Nov 03 21:03:06.590856 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.590827    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:06.605492 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.605463    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"]
Nov 03 21:03:06.674707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.674392    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:06.704708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.698996    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffeef1b-d187-4e98-a791-d47cc5155c8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr\" (UID: \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:06.704708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.699057    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffeef1b-d187-4e98-a791-d47cc5155c8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr\" (UID: \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:06.704708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.699327    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2cq4\" (UniqueName: \"kubernetes.io/projected/5ffeef1b-d187-4e98-a791-d47cc5155c8e-kube-api-access-q2cq4\") pod \"payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr\" (UID: \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:06.777001 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.776924    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2cq4\" (UniqueName: \"kubernetes.io/projected/5ffeef1b-d187-4e98-a791-d47cc5155c8e-kube-api-access-q2cq4\") pod \"payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr\" (UID: \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:06.802522 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.802484    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5g5w\" (UniqueName: \"kubernetes.io/projected/9e990412-4a9b-43dc-9b2d-dea49496ce3f-kube-api-access-s5g5w\") pod \"payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p\" (UID: \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:06.802717 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.802570    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e990412-4a9b-43dc-9b2d-dea49496ce3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p\" (UID: \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:06.904605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.904076    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g5w\" (UniqueName: \"kubernetes.io/projected/9e990412-4a9b-43dc-9b2d-dea49496ce3f-kube-api-access-s5g5w\") pod \"payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p\" (UID: \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:06.904605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.904219    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e990412-4a9b-43dc-9b2d-dea49496ce3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p\" (UID: \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:06.905523 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.905497    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e990412-4a9b-43dc-9b2d-dea49496ce3f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p\" (UID: \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:06.957762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.957725    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g5w\" (UniqueName: \"kubernetes.io/projected/9e990412-4a9b-43dc-9b2d-dea49496ce3f-kube-api-access-s5g5w\") pod \"payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p\" (UID: \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:06.990038 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:06.990007    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-a3fe47"
Nov 03 21:03:07.035930 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.035896    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:07.106416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.106392    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0e02d6-e9e6-471c-af19-8d1006d6d813-kube-api-access-4pc7h\" (UniqueName: \"kubernetes.io/projected/2c0e02d6-e9e6-471c-af19-8d1006d6d813-kube-api-access-4pc7h\") pod \"2c0e02d6-e9e6-471c-af19-8d1006d6d813\" (UID: \"2c0e02d6-e9e6-471c-af19-8d1006d6d813\") "
Nov 03 21:03:07.116458 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.116423    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0e02d6-e9e6-471c-af19-8d1006d6d813-kube-api-access-4pc7h" pod "2c0e02d6-e9e6-471c-af19-8d1006d6d813" (UID: "2c0e02d6-e9e6-471c-af19-8d1006d6d813"). InnerVolumeSpecName "kube-api-access-4pc7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:07.161140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.161032    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"]
Nov 03 21:03:07.168478 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:03:07.168439    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a255c4d_6f63_4166_afe3_7598df773d34.slice/crio-0f746bd6590eae91d33e5c5cbfdf9db3f1cfa298c0683e0d6efd8a4d490e5851 WatchSource:0}: Error finding container 0f746bd6590eae91d33e5c5cbfdf9db3f1cfa298c0683e0d6efd8a4d490e5851: Status 404 returned error can't find the container with id 0f746bd6590eae91d33e5c5cbfdf9db3f1cfa298c0683e0d6efd8a4d490e5851
Nov 03 21:03:07.209149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.209021    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pc7h\" (UniqueName: \"kubernetes.io/projected/2c0e02d6-e9e6-471c-af19-8d1006d6d813-kube-api-access-4pc7h\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:07.244639 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.244605    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:07.423796 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.423765    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"]
Nov 03 21:03:07.426792 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:03:07.426762    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffeef1b_d187_4e98_a791_d47cc5155c8e.slice/crio-48b79dcf22e799ab441c75ed5dccd8981db8753ffcf4bd16e0f3851ee4e363d0 WatchSource:0}: Error finding container 48b79dcf22e799ab441c75ed5dccd8981db8753ffcf4bd16e0f3851ee4e363d0: Status 404 returned error can't find the container with id 48b79dcf22e799ab441c75ed5dccd8981db8753ffcf4bd16e0f3851ee4e363d0
Nov 03 21:03:07.499909 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.499870    5196 generic.go:356] "Generic (PLEG): container finished" podID="2c0e02d6-e9e6-471c-af19-8d1006d6d813" containerID="1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf" exitCode=137
Nov 03 21:03:07.500120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.499988    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-a3fe47"
Nov 03 21:03:07.501811 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.501773    5196 scope.go:122] "RemoveContainer" containerID="1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf"
Nov 03 21:03:07.502721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.502686    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv" event={"ID":"8a255c4d-6f63-4166-afe3-7598df773d34","Type":"ContainerStarted","Data":"0f746bd6590eae91d33e5c5cbfdf9db3f1cfa298c0683e0d6efd8a4d490e5851"}
Nov 03 21:03:07.505429 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.505405    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr" event={"ID":"5ffeef1b-d187-4e98-a791-d47cc5155c8e","Type":"ContainerStarted","Data":"48b79dcf22e799ab441c75ed5dccd8981db8753ffcf4bd16e0f3851ee4e363d0"}
Nov 03 21:03:07.521384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.521358    5196 scope.go:122] "RemoveContainer" containerID="1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf"
Nov 03 21:03:07.521719 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:03:07.521699    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf\": container with ID starting with 1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf not found: ID does not exist" containerID="1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf"
Nov 03 21:03:07.521763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.521729    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf"} err="failed to get container status \"1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf\": rpc error: code = NotFound desc = could not find container \"1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf\": container with ID starting with 1d4725ca3d1721bb7a913d41df799ed8c1a594fa2d13dbce02f06d9fcd1c94cf not found: ID does not exist"
Nov 03 21:03:07.566583 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:03:07.566538    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-a3fe47\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="2c0e02d6-e9e6-471c-af19-8d1006d6d813" pod="opendatahub/test-curl-7280-a3fe47"
Nov 03 21:03:07.726161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:07.726134    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"]
Nov 03 21:03:07.729891 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:03:07.729862    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e990412_4a9b_43dc_9b2d_dea49496ce3f.slice/crio-e94da3df2009fb6007e0034591e1ef5e39bb6a6c23492ae1f1dcc8b7ebc57bdc WatchSource:0}: Error finding container e94da3df2009fb6007e0034591e1ef5e39bb6a6c23492ae1f1dcc8b7ebc57bdc: Status 404 returned error can't find the container with id e94da3df2009fb6007e0034591e1ef5e39bb6a6c23492ae1f1dcc8b7ebc57bdc
Nov 03 21:03:08.111201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.111157    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0e02d6-e9e6-471c-af19-8d1006d6d813" path="/var/lib/kubelet/pods/2c0e02d6-e9e6-471c-af19-8d1006d6d813/volumes"
Nov 03 21:03:08.553391 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.553336    5196 generic.go:356] "Generic (PLEG): container finished" podID="8a255c4d-6f63-4166-afe3-7598df773d34" containerID="5e4b81e9b07aab53ada191281f8d643975b38fafdd167e47c490618e781b72e6" exitCode=0
Nov 03 21:03:08.562684 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.553904    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv" event={"ID":"8a255c4d-6f63-4166-afe3-7598df773d34","Type":"ContainerDied","Data":"5e4b81e9b07aab53ada191281f8d643975b38fafdd167e47c490618e781b72e6"}
Nov 03 21:03:08.566392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.566349    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p" event={"ID":"9e990412-4a9b-43dc-9b2d-dea49496ce3f","Type":"ContainerStarted","Data":"765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b"}
Nov 03 21:03:08.566662 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.566645    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p" event={"ID":"9e990412-4a9b-43dc-9b2d-dea49496ce3f","Type":"ContainerStarted","Data":"e94da3df2009fb6007e0034591e1ef5e39bb6a6c23492ae1f1dcc8b7ebc57bdc"}
Nov 03 21:03:08.567508 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.567447    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:08.570527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.570473    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr" event={"ID":"5ffeef1b-d187-4e98-a791-d47cc5155c8e","Type":"ContainerStarted","Data":"98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac"}
Nov 03 21:03:08.570843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.570805    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:08.782872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.782758    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr" podStartSLOduration=2.782729227 podStartE2EDuration="2.782729227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:08.778944548 +0000 UTC m=+2028.782903296" watchObservedRunningTime="2026-11-03 21:03:08.782729227 +0000 UTC m=+2028.786687950"
Nov 03 21:03:08.902986 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:08.899286    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p" podStartSLOduration=2.899256587 podStartE2EDuration="2.899256587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:08.896434258 +0000 UTC m=+2028.900392980" watchObservedRunningTime="2026-11-03 21:03:08.899256587 +0000 UTC m=+2028.903215342"
Nov 03 21:03:09.767189 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.767163    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:09.854641 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.854556    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-xks-ca\") pod \"8a255c4d-6f63-4166-afe3-7598df773d34\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") "
Nov 03 21:03:09.854815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.854659    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-openshift-ca\") pod \"8a255c4d-6f63-4166-afe3-7598df773d34\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") "
Nov 03 21:03:09.854815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.854778    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a255c4d-6f63-4166-afe3-7598df773d34-kube-api-access-c8gcx\" (UniqueName: \"kubernetes.io/projected/8a255c4d-6f63-4166-afe3-7598df773d34-kube-api-access-c8gcx\") pod \"8a255c4d-6f63-4166-afe3-7598df773d34\" (UID: \"8a255c4d-6f63-4166-afe3-7598df773d34\") "
Nov 03 21:03:09.855030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.855006    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-openshift-ca" pod "8a255c4d-6f63-4166-afe3-7598df773d34" (UID: "8a255c4d-6f63-4166-afe3-7598df773d34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:09.855396 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.855373    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:09.856978 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.856951    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-xks-ca" pod "8a255c4d-6f63-4166-afe3-7598df773d34" (UID: "8a255c4d-6f63-4166-afe3-7598df773d34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:09.857072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.857020    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a255c4d-6f63-4166-afe3-7598df773d34-kube-api-access-c8gcx" pod "8a255c4d-6f63-4166-afe3-7598df773d34" (UID: "8a255c4d-6f63-4166-afe3-7598df773d34"). InnerVolumeSpecName "kube-api-access-c8gcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:09.957449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.957410    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gcx\" (UniqueName: \"kubernetes.io/projected/8a255c4d-6f63-4166-afe3-7598df773d34-kube-api-access-c8gcx\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:09.957449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:09.957448    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a255c4d-6f63-4166-afe3-7598df773d34-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.601478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:10.601447    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"
Nov 03 21:03:10.601628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:10.601478    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv" event={"ID":"8a255c4d-6f63-4166-afe3-7598df773d34","Type":"ContainerDied","Data":"0f746bd6590eae91d33e5c5cbfdf9db3f1cfa298c0683e0d6efd8a4d490e5851"}
Nov 03 21:03:10.601628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:10.601561    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f746bd6590eae91d33e5c5cbfdf9db3f1cfa298c0683e0d6efd8a4d490e5851"
Nov 03 21:03:10.915733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:10.915646    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:11.195533 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:11.195447    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:11.365766 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:11.365717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:11.515694 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:11.515606    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:15.278505 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:15.278467    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:15.495354 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:15.495214    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:16.415046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:16.414999    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:16.514163 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:16.514124    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"]
Nov 03 21:03:16.615537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:16.615495    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"]
Nov 03 21:03:16.894811 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:16.894768    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:17.057160 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:17.057121    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:18.504134 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:03:18.504050    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:03:19.588688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:19.588657    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:03:19.589148 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:19.588738    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:03:21.514713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.514675    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n" podUID="c0e0e8e5-5d4e-4237-bbe4-a055a747ae58" containerName="payload-processing" containerID="cri-o://76987d64594c929acdfedf0c154e5c7d9d85ee6f2852ee28890215e472e9a93d" gracePeriod=25
Nov 03 21:03:21.616014 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.615975    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s" podUID="9fc0d76f-69d0-4539-8989-f58256e9457e" containerName="payload-pre-processing" containerID="cri-o://ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2" gracePeriod=25
Nov 03 21:03:21.720028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.719992    5196 generic.go:356] "Generic (PLEG): container finished" podID="c0e0e8e5-5d4e-4237-bbe4-a055a747ae58" containerID="76987d64594c929acdfedf0c154e5c7d9d85ee6f2852ee28890215e472e9a93d" exitCode=0
Nov 03 21:03:21.720257 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.720079    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n" event={"ID":"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58","Type":"ContainerDied","Data":"76987d64594c929acdfedf0c154e5c7d9d85ee6f2852ee28890215e472e9a93d"}
Nov 03 21:03:21.732444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.732070    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:21.878166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.878134    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 21:03:21.965575 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.965542    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-kube-api-access-7q8gd\" (UniqueName: \"kubernetes.io/projected/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-kube-api-access-7q8gd\") pod \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\" (UID: \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\") "
Nov 03 21:03:21.966146 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.965642    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-plugins-config-volume\") pod \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\" (UID: \"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58\") "
Nov 03 21:03:21.966404 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.966375    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-plugins-config-volume" pod "c0e0e8e5-5d4e-4237-bbe4-a055a747ae58" (UID: "c0e0e8e5-5d4e-4237-bbe4-a055a747ae58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:21.966659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.966639    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:21.968031 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.968009    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-kube-api-access-7q8gd" pod "c0e0e8e5-5d4e-4237-bbe4-a055a747ae58" (UID: "c0e0e8e5-5d4e-4237-bbe4-a055a747ae58"). InnerVolumeSpecName "kube-api-access-7q8gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:21.988900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:21.988873    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 21:03:22.067295 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.067261    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc0d76f-69d0-4539-8989-f58256e9457e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc0d76f-69d0-4539-8989-f58256e9457e-plugins-config-volume\") pod \"9fc0d76f-69d0-4539-8989-f58256e9457e\" (UID: \"9fc0d76f-69d0-4539-8989-f58256e9457e\") "
Nov 03 21:03:22.067540 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.067525    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc0d76f-69d0-4539-8989-f58256e9457e-kube-api-access-htfsd\" (UniqueName: \"kubernetes.io/projected/9fc0d76f-69d0-4539-8989-f58256e9457e-kube-api-access-htfsd\") pod \"9fc0d76f-69d0-4539-8989-f58256e9457e\" (UID: \"9fc0d76f-69d0-4539-8989-f58256e9457e\") "
Nov 03 21:03:22.067632 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.067604    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc0d76f-69d0-4539-8989-f58256e9457e-plugins-config-volume" pod "9fc0d76f-69d0-4539-8989-f58256e9457e" (UID: "9fc0d76f-69d0-4539-8989-f58256e9457e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:22.068064 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.068049    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q8gd\" (UniqueName: \"kubernetes.io/projected/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58-kube-api-access-7q8gd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:22.068133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.068067    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc0d76f-69d0-4539-8989-f58256e9457e-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:22.069808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.069787    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc0d76f-69d0-4539-8989-f58256e9457e-kube-api-access-htfsd" pod "9fc0d76f-69d0-4539-8989-f58256e9457e" (UID: "9fc0d76f-69d0-4539-8989-f58256e9457e"). InnerVolumeSpecName "kube-api-access-htfsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:22.169436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.169405    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htfsd\" (UniqueName: \"kubernetes.io/projected/9fc0d76f-69d0-4539-8989-f58256e9457e-kube-api-access-htfsd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:22.730340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.730303    5196 generic.go:356] "Generic (PLEG): container finished" podID="9fc0d76f-69d0-4539-8989-f58256e9457e" containerID="ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2" exitCode=0
Nov 03 21:03:22.730815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.730354    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"
Nov 03 21:03:22.730815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.730392    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s" event={"ID":"9fc0d76f-69d0-4539-8989-f58256e9457e","Type":"ContainerDied","Data":"ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2"}
Nov 03 21:03:22.730815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.730447    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s" event={"ID":"9fc0d76f-69d0-4539-8989-f58256e9457e","Type":"ContainerDied","Data":"0fa3c3b1784b356ad5da8b89c4d8d9184c61ceb2c7a275b4840e71891b3f2e7b"}
Nov 03 21:03:22.730815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.730476    5196 scope.go:122] "RemoveContainer" containerID="ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2"
Nov 03 21:03:22.732237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.732219    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"
Nov 03 21:03:22.732351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.732234    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n" event={"ID":"c0e0e8e5-5d4e-4237-bbe4-a055a747ae58","Type":"ContainerDied","Data":"295b8a44c4e9f6ca9052f627c5c507556e929d8d5c28a013add1e1153b4560a1"}
Nov 03 21:03:22.744373 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.744339    5196 scope.go:122] "RemoveContainer" containerID="ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2"
Nov 03 21:03:22.744712 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:03:22.744686    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2\": container with ID starting with ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2 not found: ID does not exist" containerID="ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2"
Nov 03 21:03:22.744789 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.744720    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2"} err="failed to get container status \"ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2\": rpc error: code = NotFound desc = could not find container \"ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2\": container with ID starting with ac08e7447184a7154b89132da2c0154e818099c5cb735aa8faecdc89f3adb2a2 not found: ID does not exist"
Nov 03 21:03:22.744789 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.744738    5196 scope.go:122] "RemoveContainer" containerID="76987d64594c929acdfedf0c154e5c7d9d85ee6f2852ee28890215e472e9a93d"
Nov 03 21:03:22.886939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.886872    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"]
Nov 03 21:03:22.934596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:22.934554    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-211d68bf-77867cff8c-zf74s"]
Nov 03 21:03:23.036375 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:23.036339    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"]
Nov 03 21:03:23.049665 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:23.049619    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-211d68bf-5f4fc56dc-9kx8n"]
Nov 03 21:03:24.111766 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:24.111730    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc0d76f-69d0-4539-8989-f58256e9457e" path="/var/lib/kubelet/pods/9fc0d76f-69d0-4539-8989-f58256e9457e/volumes"
Nov 03 21:03:24.112240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:24.112225    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e0e8e5-5d4e-4237-bbe4-a055a747ae58" path="/var/lib/kubelet/pods/c0e0e8e5-5d4e-4237-bbe4-a055a747ae58/volumes"
Nov 03 21:03:25.998796 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:25.998762    5196 scope.go:122] "RemoveContainer" containerID="60bdb26110bb3b40e6135c87f31fa350559bbcf1727736d600633a1d1b9240d7"
Nov 03 21:03:26.699162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:26.699059    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:28.735764 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:28.735726    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:03:29.980712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:29.980663    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:31.452658 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:31.452609    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:36.200694 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:36.200662    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:39.121213 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.121178    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"]
Nov 03 21:03:39.122204 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122185    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c0e0e8e5-5d4e-4237-bbe4-a055a747ae58" containerName="payload-processing"
Nov 03 21:03:39.122204 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122205    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e0e8e5-5d4e-4237-bbe4-a055a747ae58" containerName="payload-processing"
Nov 03 21:03:39.122357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122222    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9fc0d76f-69d0-4539-8989-f58256e9457e" containerName="payload-pre-processing"
Nov 03 21:03:39.122357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122227    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc0d76f-69d0-4539-8989-f58256e9457e" containerName="payload-pre-processing"
Nov 03 21:03:39.122357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122238    5196 cpu_manager.go:414] "containerMap: removing container" podUID="8a255c4d-6f63-4166-afe3-7598df773d34" containerName="revoke-keys"
Nov 03 21:03:39.122357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122243    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a255c4d-6f63-4166-afe3-7598df773d34" containerName="revoke-keys"
Nov 03 21:03:39.122486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122377    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e0e8e5-5d4e-4237-bbe4-a055a747ae58" containerName="payload-processing"
Nov 03 21:03:39.122486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122386    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc0d76f-69d0-4539-8989-f58256e9457e" containerName="payload-pre-processing"
Nov 03 21:03:39.122486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.122394    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a255c4d-6f63-4166-afe3-7598df773d34" containerName="revoke-keys"
Nov 03 21:03:39.126591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.126568    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.175680 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.175647    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-20d356-openshift-default-dockercfg-fndvt\""
Nov 03 21:03:39.227324 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.227281    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.227653 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.227627    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.227821 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.227793    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.227949 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.227846    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.227949 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.227935    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66w2\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-kube-api-access-v66w2\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.228063 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.228028    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.230197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.228060    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.230197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.228177    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.230197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.228392    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.317123 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.317068    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"]
Nov 03 21:03:39.329695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.329656    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.329924 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.329841    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.330016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.329953    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.330467 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.330434    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.330649 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.330462    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.330777 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.330759    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.330930 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.330912    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.331250 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.331227    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66w2\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-kube-api-access-v66w2\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.331775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.331443    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.331946 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.331919    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.332074 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.331964    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.332074 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.332020    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.332575 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.332554    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.332702 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.332676    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.332986 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.332963    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.342778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.342748    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.385328 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.385247    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.385478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.385379    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66w2\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-kube-api-access-v66w2\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.442285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.442243    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:39.887108 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.887058    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"]
Nov 03 21:03:39.934043 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:03:39.934008    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8d15ba_871f_4bd4_bbbb_a2f2d40ae93c.slice/crio-34e7057308a4fab42e69bdfaa33866c00fcef8c5e3895638548bcf2ff8cc09a8 WatchSource:0}: Error finding container 34e7057308a4fab42e69bdfaa33866c00fcef8c5e3895638548bcf2ff8cc09a8: Status 404 returned error can't find the container with id 34e7057308a4fab42e69bdfaa33866c00fcef8c5e3895638548bcf2ff8cc09a8
Nov 03 21:03:39.937026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.936988    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:03:39.937146 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.937080    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:03:39.937212 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:39.937191    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:03:40.897386 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:40.897340    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" event={"ID":"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c","Type":"ContainerStarted","Data":"889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba"}
Nov 03 21:03:40.897828 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:40.897454    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" event={"ID":"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c","Type":"ContainerStarted","Data":"34e7057308a4fab42e69bdfaa33866c00fcef8c5e3895638548bcf2ff8cc09a8"}
Nov 03 21:03:41.080144 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:41.080071    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" podStartSLOduration=2.080055052 podStartE2EDuration="2.080055052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:41.077348968 +0000 UTC m=+2061.081307711" watchObservedRunningTime="2026-11-03 21:03:41.080055052 +0000 UTC m=+2061.084013775"
Nov 03 21:03:41.124156 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:41.115675    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:41.458122 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:41.457994    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:41.461720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:41.461684    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:03:41.461988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:41.461958    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:03:42.455230 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:42.454834    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:03:42.455230 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:42.454899    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:03:43.443609 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:43.443566    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:03:43.443922 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:43.443887    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:03:44.453120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:44.451525    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:03:44.453120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:44.451608    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:03:44.888078 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:44.888042    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"]
Nov 03 21:03:44.888462 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:44.888343    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" containerID="cri-o://b0de665c74aa3eb99d57f3801e01611b57927786bcea71b4f72d54332c32fbcc" gracePeriod=30
Nov 03 21:03:44.981328 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:44.981286    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"]
Nov 03 21:03:44.981543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:44.981507    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy" containerID="cri-o://ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c" gracePeriod=30
Nov 03 21:03:44.987504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:44.987473    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"]
Nov 03 21:03:44.987745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:44.987693    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" containerID="cri-o://47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08" gracePeriod=30
Nov 03 21:03:45.452597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:45.452555    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:03:45.452902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:45.452869    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:03:46.389801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.389759    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"]
Nov 03 21:03:46.394742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.394718    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.457122 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.449499    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:03:46.460020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.459986    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:46.469701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.461995    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.469701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.466953    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.469701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.467061    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpz92\" (UniqueName: \"kubernetes.io/projected/4a1a1a69-6464-4831-bae6-5f926c21585f-kube-api-access-tpz92\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.469701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.469203    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 03 21:03:46.469701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.469266    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 03 21:03:46.469701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.469588    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"]
Nov 03 21:03:46.558819 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.558777    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:46.568513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.568479    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.568743 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.568587    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpz92\" (UniqueName: \"kubernetes.io/projected/4a1a1a69-6464-4831-bae6-5f926c21585f-kube-api-access-tpz92\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.568743 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.568685    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.569252 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.569232    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.571458 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.571428    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.625333 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.625286    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpz92\" (UniqueName: \"kubernetes.io/projected/4a1a1a69-6464-4831-bae6-5f926c21585f-kube-api-access-tpz92\") pod \"maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:46.712538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:46.712453    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:47.105719 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.105651    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"]
Nov 03 21:03:47.121373 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:03:47.121337    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1a1a69_6464_4831_bae6_5f926c21585f.slice/crio-832da73ed8e04ef57ec35092d1f837d225ba2cf7d9730cd1b7ebd4d144862418 WatchSource:0}: Error finding container 832da73ed8e04ef57ec35092d1f837d225ba2cf7d9730cd1b7ebd4d144862418: Status 404 returned error can't find the container with id 832da73ed8e04ef57ec35092d1f837d225ba2cf7d9730cd1b7ebd4d144862418
Nov 03 21:03:47.361886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.361806    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:47.448383 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.448351    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:47.448997 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.448956    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:47.450242 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.450216    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:03:47.704001 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.703954    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:03:47.704316 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.704282    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:03:47.988220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.988126    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w" event={"ID":"4a1a1a69-6464-4831-bae6-5f926c21585f","Type":"ContainerStarted","Data":"b5600df1c8e9f98458a1958e90cd5a192da16c79527c72741d0d970a392ec48d"}
Nov 03 21:03:47.988220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:47.988198    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w" event={"ID":"4a1a1a69-6464-4831-bae6-5f926c21585f","Type":"ContainerStarted","Data":"832da73ed8e04ef57ec35092d1f837d225ba2cf7d9730cd1b7ebd4d144862418"}
Nov 03 21:03:48.105589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:48.105500    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w" podStartSLOduration=2.10548108 podStartE2EDuration="2.10548108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:48.096757441 +0000 UTC m=+2068.100716183" watchObservedRunningTime="2026-11-03 21:03:48.10548108 +0000 UTC m=+2068.109439806"
Nov 03 21:03:48.488014 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:03:48.487976    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:03:49.000073 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:49.000038    5196 generic.go:356] "Generic (PLEG): container finished" podID="4a1a1a69-6464-4831-bae6-5f926c21585f" containerID="b5600df1c8e9f98458a1958e90cd5a192da16c79527c72741d0d970a392ec48d" exitCode=0
Nov 03 21:03:49.000290 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:49.000129    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w" event={"ID":"4a1a1a69-6464-4831-bae6-5f926c21585f","Type":"ContainerDied","Data":"b5600df1c8e9f98458a1958e90cd5a192da16c79527c72741d0d970a392ec48d"}
Nov 03 21:03:49.222774 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:49.222736    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:03:49.222974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:49.222809    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:03:49.931961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:49.931929    5196 patch_prober.go:29] interesting pod/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:03:49.932317 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:49.931986    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:03:50.019602 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.019572    5196 generic.go:356] "Generic (PLEG): container finished" podID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerID="b0de665c74aa3eb99d57f3801e01611b57927786bcea71b4f72d54332c32fbcc" exitCode=0
Nov 03 21:03:50.019704 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.019636    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" event={"ID":"5a185b07-d07c-4a15-ba9d-adddb724d421","Type":"ContainerDied","Data":"b0de665c74aa3eb99d57f3801e01611b57927786bcea71b4f72d54332c32fbcc"}
Nov 03 21:03:50.270816 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.270470    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 21:03:50.276298 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.276255    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:50.335815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.335770    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-openshift-ca\") pod \"4a1a1a69-6464-4831-bae6-5f926c21585f\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") "
Nov 03 21:03:50.336045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.335830    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-token\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-token\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.336045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.335869    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-socket\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.336045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.335894    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-envoy\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.336045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.335934    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1a1a69-6464-4831-bae6-5f926c21585f-kube-api-access-tpz92\" (UniqueName: \"kubernetes.io/projected/4a1a1a69-6464-4831-bae6-5f926c21585f-kube-api-access-tpz92\") pod \"4a1a1a69-6464-4831-bae6-5f926c21585f\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") "
Nov 03 21:03:50.336322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.336048    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-certs\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.336322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.336082    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a185b07-d07c-4a15-ba9d-adddb724d421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a185b07-d07c-4a15-ba9d-adddb724d421-istiod-ca-cert\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.336322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.336155    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-credential-socket\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.336322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.336287    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-data\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.336513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.336320    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-xks-ca\") pod \"4a1a1a69-6464-4831-bae6-5f926c21585f\" (UID: \"4a1a1a69-6464-4831-bae6-5f926c21585f\") "
Nov 03 21:03:50.336513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.336412    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-podinfo\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.336614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.336509    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-kube-api-access-m66x4\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-kube-api-access-m66x4\") pod \"5a185b07-d07c-4a15-ba9d-adddb724d421\" (UID: \"5a185b07-d07c-4a15-ba9d-adddb724d421\") "
Nov 03 21:03:50.338160 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.338114    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-certs" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.338297 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.338201    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-socket" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.338297 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.338221    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-data" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.338603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.338578    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-openshift-ca" pod "4a1a1a69-6464-4831-bae6-5f926c21585f" (UID: "4a1a1a69-6464-4831-bae6-5f926c21585f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:50.338664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.338644    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a185b07-d07c-4a15-ba9d-adddb724d421-istiod-ca-cert" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:50.338855 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.338829    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-credential-socket" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.341444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.341366    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-podinfo" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:03:50.343080 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.343054    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-token" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:50.343080 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.343061    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-kube-api-access-m66x4" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "kube-api-access-m66x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:50.343930 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.343887    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-xks-ca" pod "4a1a1a69-6464-4831-bae6-5f926c21585f" (UID: "4a1a1a69-6464-4831-bae6-5f926c21585f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:50.345427 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.345398    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-envoy" pod "5a185b07-d07c-4a15-ba9d-adddb724d421" (UID: "5a185b07-d07c-4a15-ba9d-adddb724d421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.346500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.346472    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1a1a69-6464-4831-bae6-5f926c21585f-kube-api-access-tpz92" pod "4a1a1a69-6464-4831-bae6-5f926c21585f" (UID: "4a1a1a69-6464-4831-bae6-5f926c21585f"). InnerVolumeSpecName "kube-api-access-tpz92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:50.427662 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.427618    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 21:03:50.430970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.430936    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 21:03:50.438553 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438526    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438553 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438554    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438564    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438573    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438581    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m66x4\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-kube-api-access-m66x4\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438591    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a1a69-6464-4831-bae6-5f926c21585f-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438600    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438609    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438618    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438628    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpz92\" (UniqueName: \"kubernetes.io/projected/4a1a1a69-6464-4831-bae6-5f926c21585f-kube-api-access-tpz92\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438636    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a185b07-d07c-4a15-ba9d-adddb724d421-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.438708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.438644    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a185b07-d07c-4a15-ba9d-adddb724d421-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.539724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.539633    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7aefef8-4211-4e77-9949-2223b25363ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7aefef8-4211-4e77-9949-2223b25363ca-istiod-ca-cert\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.539724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.539687    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-token\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-token\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.539724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.539721    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cdfe258-b7fb-41a0-b137-9740277878d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdfe258-b7fb-41a0-b137-9740277878d5-istiod-ca-cert\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.540027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.539796    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-istio-token\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-istio-token\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.540027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.539856    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-podinfo\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.540027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.539884    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-certs\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.540027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.539926    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-credential-socket\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.540027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540001    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-kube-api-access-7tlq5\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-kube-api-access-7tlq5\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.540311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540078    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-envoy\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.540311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540143    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7aefef8-4211-4e77-9949-2223b25363ca-istiod-ca-cert" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:50.540311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540157    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-socket\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.540311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540243    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-certs\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.540543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540340    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7aefef8-4211-4e77-9949-2223b25363ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7aefef8-4211-4e77-9949-2223b25363ca-istio-podinfo\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.540543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540370    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-envoy\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.540543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540399    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-data\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.540543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540426    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-data\") pod \"b7aefef8-4211-4e77-9949-2223b25363ca\" (UID: \"b7aefef8-4211-4e77-9949-2223b25363ca\") "
Nov 03 21:03:50.540543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540467    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-certs" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.540543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540526    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-kube-api-access-qlpqs\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-kube-api-access-qlpqs\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.540841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540587    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-credential-socket\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.540841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540675    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-socket\") pod \"5cdfe258-b7fb-41a0-b137-9740277878d5\" (UID: \"5cdfe258-b7fb-41a0-b137-9740277878d5\") "
Nov 03 21:03:50.540935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.540915    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdfe258-b7fb-41a0-b137-9740277878d5-istiod-ca-cert" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:50.541339 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.541277    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-credential-socket" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.542203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.541527    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-socket" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.542203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.541537    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-data" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.542203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.541901    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-data" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.542203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.541987    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-credential-socket" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.542203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542016    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-socket" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.542203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542153    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-certs" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.542781 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542755    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.542882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542788    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.542882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542802    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.542882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542813    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.542882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542828    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.542882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542840    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.542882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542853    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7aefef8-4211-4e77-9949-2223b25363ca-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.542882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542865    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdfe258-b7fb-41a0-b137-9740277878d5-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.542882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542878    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.543485 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.542894    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.543485 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.543153    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-token" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:50.544998 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.544973    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-istio-token" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:50.545358 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.545312    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-kube-api-access-qlpqs" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "kube-api-access-qlpqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:50.545761 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.545727    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-kube-api-access-7tlq5" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "kube-api-access-7tlq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:50.545901 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.545880    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-envoy" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.545978 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.545878    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7aefef8-4211-4e77-9949-2223b25363ca-istio-podinfo" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:03:50.546591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.546572    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-envoy" pod "b7aefef8-4211-4e77-9949-2223b25363ca" (UID: "b7aefef8-4211-4e77-9949-2223b25363ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:50.547046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.547029    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-podinfo" pod "5cdfe258-b7fb-41a0-b137-9740277878d5" (UID: "5cdfe258-b7fb-41a0-b137-9740277878d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:03:50.645241 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.645193    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlpqs\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-kube-api-access-qlpqs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.645241 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.645237    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.645551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.645253    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.645551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.645278    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.645551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.645293    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tlq5\" (UniqueName: \"kubernetes.io/projected/b7aefef8-4211-4e77-9949-2223b25363ca-kube-api-access-7tlq5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.645551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.645306    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdfe258-b7fb-41a0-b137-9740277878d5-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.645551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.645320    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7aefef8-4211-4e77-9949-2223b25363ca-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.645551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:50.645336    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7aefef8-4211-4e77-9949-2223b25363ca-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:51.040762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.040728    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"
Nov 03 21:03:51.042909 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.042483    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w" event={"ID":"4a1a1a69-6464-4831-bae6-5f926c21585f","Type":"ContainerDied","Data":"832da73ed8e04ef57ec35092d1f837d225ba2cf7d9730cd1b7ebd4d144862418"}
Nov 03 21:03:51.042909 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.042544    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="832da73ed8e04ef57ec35092d1f837d225ba2cf7d9730cd1b7ebd4d144862418"
Nov 03 21:03:51.044402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.044375    5196 generic.go:356] "Generic (PLEG): container finished" podID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerID="ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c" exitCode=0
Nov 03 21:03:51.044547 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.044534    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" event={"ID":"5cdfe258-b7fb-41a0-b137-9740277878d5","Type":"ContainerDied","Data":"ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c"}
Nov 03 21:03:51.044593 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.044572    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q" event={"ID":"5cdfe258-b7fb-41a0-b137-9740277878d5","Type":"ContainerDied","Data":"ec1e5368712ef6ee35fac3604d713975390448f90142ab85a4d31d01046ff469"}
Nov 03 21:03:51.044627 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.044594    5196 scope.go:122] "RemoveContainer" containerID="ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c"
Nov 03 21:03:51.044727 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.044716    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"
Nov 03 21:03:51.049557 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.049526    5196 generic.go:356] "Generic (PLEG): container finished" podID="b7aefef8-4211-4e77-9949-2223b25363ca" containerID="47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08" exitCode=0
Nov 03 21:03:51.049670 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.049613    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" event={"ID":"b7aefef8-4211-4e77-9949-2223b25363ca","Type":"ContainerDied","Data":"47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08"}
Nov 03 21:03:51.049727 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.049675    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp" event={"ID":"b7aefef8-4211-4e77-9949-2223b25363ca","Type":"ContainerDied","Data":"5947b768a2c381b323de3a8d609c5dc522f465c660e96dfc626aa86a748670d3"}
Nov 03 21:03:51.049871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.049842    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"
Nov 03 21:03:51.057530 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.056815    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24" event={"ID":"5a185b07-d07c-4a15-ba9d-adddb724d421","Type":"ContainerDied","Data":"b681f963dc92ea65cae0a1fb1d2f89a3816df9a784beec0c6965872891a7ec19"}
Nov 03 21:03:51.060605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.060224    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"
Nov 03 21:03:51.078552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.078524    5196 scope.go:122] "RemoveContainer" containerID="ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c"
Nov 03 21:03:51.078939 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:03:51.078917    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c\": container with ID starting with ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c not found: ID does not exist" containerID="ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c"
Nov 03 21:03:51.079033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.078952    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c"} err="failed to get container status \"ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c\": rpc error: code = NotFound desc = could not find container \"ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c\": container with ID starting with ae0d3c97a542e8632eef3dc36bd1fe6ca5a36d2d3b6c9696b139a70fdb8d484c not found: ID does not exist"
Nov 03 21:03:51.079033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.078978    5196 scope.go:122] "RemoveContainer" containerID="47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08"
Nov 03 21:03:51.093455 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.093429    5196 scope.go:122] "RemoveContainer" containerID="47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08"
Nov 03 21:03:51.093803 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:03:51.093765    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08\": container with ID starting with 47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08 not found: ID does not exist" containerID="47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08"
Nov 03 21:03:51.093916 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.093812    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08"} err="failed to get container status \"47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08\": rpc error: code = NotFound desc = could not find container \"47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08\": container with ID starting with 47f87267d95966e165557e6f50ac3bbfd2b82250db0174a362799faf2d35ac08 not found: ID does not exist"
Nov 03 21:03:51.093916 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.093831    5196 scope.go:122] "RemoveContainer" containerID="b0de665c74aa3eb99d57f3801e01611b57927786bcea71b4f72d54332c32fbcc"
Nov 03 21:03:51.203006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.202902    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"]
Nov 03 21:03:51.207983 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.207954    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-hs24q"]
Nov 03 21:03:51.359150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.359109    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"]
Nov 03 21:03:51.365185 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.365150    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-xk4kp"]
Nov 03 21:03:51.468412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.468325    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"]
Nov 03 21:03:51.472894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.472867    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-xks24"]
Nov 03 21:03:51.764041 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:51.763966    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:52.111833 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:52.111797    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" path="/var/lib/kubelet/pods/5a185b07-d07c-4a15-ba9d-adddb724d421/volumes"
Nov 03 21:03:52.112377 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:52.112362    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" path="/var/lib/kubelet/pods/5cdfe258-b7fb-41a0-b137-9740277878d5/volumes"
Nov 03 21:03:52.112817 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:52.112804    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" path="/var/lib/kubelet/pods/b7aefef8-4211-4e77-9949-2223b25363ca/volumes"
Nov 03 21:03:53.874701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.874666    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"]
Nov 03 21:03:53.875730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.875706    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy"
Nov 03 21:03:53.875730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.875730    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy"
Nov 03 21:03:53.875903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.875745    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy"
Nov 03 21:03:53.875903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.875754    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy"
Nov 03 21:03:53.875903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.875804    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy"
Nov 03 21:03:53.875903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.875811    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy"
Nov 03 21:03:53.875903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.875828    5196 cpu_manager.go:414] "containerMap: removing container" podUID="4a1a1a69-6464-4831-bae6-5f926c21585f" containerName="revoke-keys"
Nov 03 21:03:53.875903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.875835    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1a1a69-6464-4831-bae6-5f926c21585f" containerName="revoke-keys"
Nov 03 21:03:53.876220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.876068    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7aefef8-4211-4e77-9949-2223b25363ca" containerName="istio-proxy"
Nov 03 21:03:53.876220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.876087    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1a1a69-6464-4831-bae6-5f926c21585f" containerName="revoke-keys"
Nov 03 21:03:53.876220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.876125    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cdfe258-b7fb-41a0-b137-9740277878d5" containerName="istio-proxy"
Nov 03 21:03:53.876220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.876136    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a185b07-d07c-4a15-ba9d-adddb724d421" containerName="istio-proxy"
Nov 03 21:03:53.882015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.881990    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:53.931742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.931709    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"]
Nov 03 21:03:53.973156 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.973041    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:53.973341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:53.973174    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:03:54.015517 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.015480    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.015671 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.015641    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.015820 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.015804    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzf6z\" (UniqueName: \"kubernetes.io/projected/41ce50c4-311e-4422-a31e-80136560a0ba-kube-api-access-rzf6z\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.123265 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.123228    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf6z\" (UniqueName: \"kubernetes.io/projected/41ce50c4-311e-4422-a31e-80136560a0ba-kube-api-access-rzf6z\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.123462 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.123339    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.123512 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.123464    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.124179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.124148    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.126050 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.125999    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.177146 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.177087    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf6z\" (UniqueName: \"kubernetes.io/projected/41ce50c4-311e-4422-a31e-80136560a0ba-kube-api-access-rzf6z\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.199476 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.199448    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:54.584947 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:54.584912    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"]
Nov 03 21:03:54.590045 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:03:54.590017    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ce50c4_311e_4422_a31e_80136560a0ba.slice/crio-717baeb1b23a60a201f6921f5a1b4a68560c0ee46c0f4c0e233504e84cc171f9 WatchSource:0}: Error finding container 717baeb1b23a60a201f6921f5a1b4a68560c0ee46c0f4c0e233504e84cc171f9: Status 404 returned error can't find the container with id 717baeb1b23a60a201f6921f5a1b4a68560c0ee46c0f4c0e233504e84cc171f9
Nov 03 21:03:55.096396 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:55.096358    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm" event={"ID":"41ce50c4-311e-4422-a31e-80136560a0ba","Type":"ContainerStarted","Data":"3c5ad09d9ad339e8d295152a46e1000215742531bf2ef1a17852d557e6ea91e5"}
Nov 03 21:03:55.096801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:55.096419    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm" event={"ID":"41ce50c4-311e-4422-a31e-80136560a0ba","Type":"ContainerStarted","Data":"717baeb1b23a60a201f6921f5a1b4a68560c0ee46c0f4c0e233504e84cc171f9"}
Nov 03 21:03:55.266804 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:55.266698    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm" podStartSLOduration=2.26668409 podStartE2EDuration="2.26668409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:55.265465289 +0000 UTC m=+2075.269424035" watchObservedRunningTime="2026-11-03 21:03:55.26668409 +0000 UTC m=+2075.270642813"
Nov 03 21:03:56.238116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:56.237927    5196 generic.go:356] "Generic (PLEG): container finished" podID="41ce50c4-311e-4422-a31e-80136560a0ba" containerID="3c5ad09d9ad339e8d295152a46e1000215742531bf2ef1a17852d557e6ea91e5" exitCode=0
Nov 03 21:03:56.238665 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:56.238199    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm" event={"ID":"41ce50c4-311e-4422-a31e-80136560a0ba","Type":"ContainerDied","Data":"3c5ad09d9ad339e8d295152a46e1000215742531bf2ef1a17852d557e6ea91e5"}
Nov 03 21:03:56.872882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:56.872848    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:57.245204 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.245121    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"]
Nov 03 21:03:57.250199 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.250177    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.297661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.297612    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.297848 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.297759    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxjxp\" (UniqueName: \"kubernetes.io/projected/6b2f3547-8792-43a5-b67a-d10256e6dd1b-kube-api-access-pxjxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.297848 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.297821    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.304935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.304903    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"]
Nov 03 21:03:57.399332 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.399291    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxjxp\" (UniqueName: \"kubernetes.io/projected/6b2f3547-8792-43a5-b67a-d10256e6dd1b-kube-api-access-pxjxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.399542 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.399383    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.399707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.399687    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.400379 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.400350    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.402248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.402225    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.453493 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.453455    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxjxp\" (UniqueName: \"kubernetes.io/projected/6b2f3547-8792-43a5-b67a-d10256e6dd1b-kube-api-access-pxjxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.457550 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.457418    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:57.501265 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.501171    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ce50c4-311e-4422-a31e-80136560a0ba-kube-api-access-rzf6z\" (UniqueName: \"kubernetes.io/projected/41ce50c4-311e-4422-a31e-80136560a0ba-kube-api-access-rzf6z\") pod \"41ce50c4-311e-4422-a31e-80136560a0ba\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") "
Nov 03 21:03:57.501432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.501275    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-xks-ca\") pod \"41ce50c4-311e-4422-a31e-80136560a0ba\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") "
Nov 03 21:03:57.501432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.501322    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-openshift-ca\") pod \"41ce50c4-311e-4422-a31e-80136560a0ba\" (UID: \"41ce50c4-311e-4422-a31e-80136560a0ba\") "
Nov 03 21:03:57.501726 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.501703    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-openshift-ca" pod "41ce50c4-311e-4422-a31e-80136560a0ba" (UID: "41ce50c4-311e-4422-a31e-80136560a0ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:57.502003 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.501951    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:57.503642 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.503616    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-xks-ca" pod "41ce50c4-311e-4422-a31e-80136560a0ba" (UID: "41ce50c4-311e-4422-a31e-80136560a0ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:57.503792 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.503772    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ce50c4-311e-4422-a31e-80136560a0ba-kube-api-access-rzf6z" pod "41ce50c4-311e-4422-a31e-80136560a0ba" (UID: "41ce50c4-311e-4422-a31e-80136560a0ba"). InnerVolumeSpecName "kube-api-access-rzf6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:57.572726 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.572690    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:03:57.603732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.603696    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzf6z\" (UniqueName: \"kubernetes.io/projected/41ce50c4-311e-4422-a31e-80136560a0ba-kube-api-access-rzf6z\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:57.603732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.603735    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ce50c4-311e-4422-a31e-80136560a0ba-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:57.960132 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:57.960076    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"]
Nov 03 21:03:57.965326 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:03:57.965296    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2f3547_8792_43a5_b67a_d10256e6dd1b.slice/crio-7f822c9e4d20ddcc01980453994ed85898e7a7a98f37908a6b8074cb496ff15c WatchSource:0}: Error finding container 7f822c9e4d20ddcc01980453994ed85898e7a7a98f37908a6b8074cb496ff15c: Status 404 returned error can't find the container with id 7f822c9e4d20ddcc01980453994ed85898e7a7a98f37908a6b8074cb496ff15c
Nov 03 21:03:58.257076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:58.256993    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"
Nov 03 21:03:58.257076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:58.257021    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm" event={"ID":"41ce50c4-311e-4422-a31e-80136560a0ba","Type":"ContainerDied","Data":"717baeb1b23a60a201f6921f5a1b4a68560c0ee46c0f4c0e233504e84cc171f9"}
Nov 03 21:03:58.257587 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:58.257151    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="717baeb1b23a60a201f6921f5a1b4a68560c0ee46c0f4c0e233504e84cc171f9"
Nov 03 21:03:58.258578 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:58.258548    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn" event={"ID":"6b2f3547-8792-43a5-b67a-d10256e6dd1b","Type":"ContainerStarted","Data":"7f822c9e4d20ddcc01980453994ed85898e7a7a98f37908a6b8074cb496ff15c"}
Nov 03 21:03:59.268721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:59.268682    5196 generic.go:356] "Generic (PLEG): container finished" podID="6b2f3547-8792-43a5-b67a-d10256e6dd1b" containerID="ac33964eb02eb1917e86279c3a8c25b00084fa08b840c0b668046c1a093dc6ce" exitCode=0
Nov 03 21:03:59.269117 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:03:59.268769    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn" event={"ID":"6b2f3547-8792-43a5-b67a-d10256e6dd1b","Type":"ContainerDied","Data":"ac33964eb02eb1917e86279c3a8c25b00084fa08b840c0b668046c1a093dc6ce"}
Nov 03 21:04:00.446120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.446082    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:04:00.554606 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.554523    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-openshift-ca\") pod \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") "
Nov 03 21:04:00.554893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.554859    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-openshift-ca" pod "6b2f3547-8792-43a5-b67a-d10256e6dd1b" (UID: "6b2f3547-8792-43a5-b67a-d10256e6dd1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:00.555010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.554968    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-xks-ca\") pod \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") "
Nov 03 21:04:00.555086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.555073    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2f3547-8792-43a5-b67a-d10256e6dd1b-kube-api-access-pxjxp\" (UniqueName: \"kubernetes.io/projected/6b2f3547-8792-43a5-b67a-d10256e6dd1b-kube-api-access-pxjxp\") pod \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\" (UID: \"6b2f3547-8792-43a5-b67a-d10256e6dd1b\") "
Nov 03 21:04:00.555781 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.555758    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:00.557897 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.557871    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-xks-ca" pod "6b2f3547-8792-43a5-b67a-d10256e6dd1b" (UID: "6b2f3547-8792-43a5-b67a-d10256e6dd1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:00.557991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.557939    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2f3547-8792-43a5-b67a-d10256e6dd1b-kube-api-access-pxjxp" pod "6b2f3547-8792-43a5-b67a-d10256e6dd1b" (UID: "6b2f3547-8792-43a5-b67a-d10256e6dd1b"). InnerVolumeSpecName "kube-api-access-pxjxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:00.657829 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.657788    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b2f3547-8792-43a5-b67a-d10256e6dd1b-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:00.657829 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:00.657827    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxjxp\" (UniqueName: \"kubernetes.io/projected/6b2f3547-8792-43a5-b67a-d10256e6dd1b-kube-api-access-pxjxp\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:01.289178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.289145    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn" event={"ID":"6b2f3547-8792-43a5-b67a-d10256e6dd1b","Type":"ContainerDied","Data":"7f822c9e4d20ddcc01980453994ed85898e7a7a98f37908a6b8074cb496ff15c"}
Nov 03 21:04:01.289383 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.289181    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"
Nov 03 21:04:01.289383 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.289193    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f822c9e4d20ddcc01980453994ed85898e7a7a98f37908a6b8074cb496ff15c"
Nov 03 21:04:01.312823 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.312795    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"]
Nov 03 21:04:01.313744 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.313723    5196 cpu_manager.go:414] "containerMap: removing container" podUID="41ce50c4-311e-4422-a31e-80136560a0ba" containerName="revoke-keys"
Nov 03 21:04:01.313744 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.313743    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ce50c4-311e-4422-a31e-80136560a0ba" containerName="revoke-keys"
Nov 03 21:04:01.313921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.313770    5196 cpu_manager.go:414] "containerMap: removing container" podUID="6b2f3547-8792-43a5-b67a-d10256e6dd1b" containerName="revoke-keys"
Nov 03 21:04:01.313921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.313775    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2f3547-8792-43a5-b67a-d10256e6dd1b" containerName="revoke-keys"
Nov 03 21:04:01.313921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.313921    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ce50c4-311e-4422-a31e-80136560a0ba" containerName="revoke-keys"
Nov 03 21:04:01.314023 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.313932    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2f3547-8792-43a5-b67a-d10256e6dd1b" containerName="revoke-keys"
Nov 03 21:04:01.317747 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.317727    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.366974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.366945    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:01.368877 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.368840    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4x9\" (UniqueName: \"kubernetes.io/projected/938668d4-f1f1-4a19-8b4e-746dd01849ec-kube-api-access-pr4x9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.369219 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.369184    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.369465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.369436    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.370385 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.370357    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"]
Nov 03 21:04:01.395422 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.395397    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:04:01.471375 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.471337    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.471906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.471408    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.471906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.471585    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4x9\" (UniqueName: \"kubernetes.io/projected/938668d4-f1f1-4a19-8b4e-746dd01849ec-kube-api-access-pr4x9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.472193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.472169    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.474028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.474009    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.525718 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.525686    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4x9\" (UniqueName: \"kubernetes.io/projected/938668d4-f1f1-4a19-8b4e-746dd01849ec-kube-api-access-pr4x9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.633160 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.633085    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:01.998606 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:01.998513    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:02.023825 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:02.023761    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"]
Nov 03 21:04:02.299178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:02.299072    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9" event={"ID":"938668d4-f1f1-4a19-8b4e-746dd01849ec","Type":"ContainerStarted","Data":"50fe32e20b47621ef5b005b756148ddd0ce7a9ab4d042a6067f0b4fb19cc2201"}
Nov 03 21:04:03.311339 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:03.311242    5196 generic.go:356] "Generic (PLEG): container finished" podID="938668d4-f1f1-4a19-8b4e-746dd01849ec" containerID="00fbe0e3e528257ac8124a379906ca2b2eea70c044dd4f94562b4888f63e43b0" exitCode=0
Nov 03 21:04:03.311725 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:03.311336    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9" event={"ID":"938668d4-f1f1-4a19-8b4e-746dd01849ec","Type":"ContainerDied","Data":"00fbe0e3e528257ac8124a379906ca2b2eea70c044dd4f94562b4888f63e43b0"}
Nov 03 21:04:04.522791 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.522765    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:04.617486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.617397    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-openshift-ca\") pod \"938668d4-f1f1-4a19-8b4e-746dd01849ec\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") "
Nov 03 21:04:04.617678 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.617655    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-xks-ca\") pod \"938668d4-f1f1-4a19-8b4e-746dd01849ec\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") "
Nov 03 21:04:04.617808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.617790    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938668d4-f1f1-4a19-8b4e-746dd01849ec-kube-api-access-pr4x9\" (UniqueName: \"kubernetes.io/projected/938668d4-f1f1-4a19-8b4e-746dd01849ec-kube-api-access-pr4x9\") pod \"938668d4-f1f1-4a19-8b4e-746dd01849ec\" (UID: \"938668d4-f1f1-4a19-8b4e-746dd01849ec\") "
Nov 03 21:04:04.617870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.617835    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-openshift-ca" pod "938668d4-f1f1-4a19-8b4e-746dd01849ec" (UID: "938668d4-f1f1-4a19-8b4e-746dd01849ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:04.618397 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.618376    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.620037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.620011    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938668d4-f1f1-4a19-8b4e-746dd01849ec-kube-api-access-pr4x9" pod "938668d4-f1f1-4a19-8b4e-746dd01849ec" (UID: "938668d4-f1f1-4a19-8b4e-746dd01849ec"). InnerVolumeSpecName "kube-api-access-pr4x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.620037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.620029    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-xks-ca" pod "938668d4-f1f1-4a19-8b4e-746dd01849ec" (UID: "938668d4-f1f1-4a19-8b4e-746dd01849ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:04.719548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.719510    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938668d4-f1f1-4a19-8b4e-746dd01849ec-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.719548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:04.719542    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr4x9\" (UniqueName: \"kubernetes.io/projected/938668d4-f1f1-4a19-8b4e-746dd01849ec-kube-api-access-pr4x9\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:05.043724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:05.043634    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:05.330148 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:05.330111    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"
Nov 03 21:04:05.330148 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:05.330132    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9" event={"ID":"938668d4-f1f1-4a19-8b4e-746dd01849ec","Type":"ContainerDied","Data":"50fe32e20b47621ef5b005b756148ddd0ce7a9ab4d042a6067f0b4fb19cc2201"}
Nov 03 21:04:05.330376 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:05.330188    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50fe32e20b47621ef5b005b756148ddd0ce7a9ab4d042a6067f0b4fb19cc2201"
Nov 03 21:04:06.496119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:06.495452    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:06.653537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:06.653218    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:07.332511 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:07.332475    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:08.981444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:08.981407    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:09.190127 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.188365    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:09.355538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.355503    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"]
Nov 03 21:04:09.356367 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.356349    5196 cpu_manager.go:414] "containerMap: removing container" podUID="938668d4-f1f1-4a19-8b4e-746dd01849ec" containerName="revoke-keys"
Nov 03 21:04:09.356367 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.356367    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938668d4-f1f1-4a19-8b4e-746dd01849ec" containerName="revoke-keys"
Nov 03 21:04:09.356516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.356507    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="938668d4-f1f1-4a19-8b4e-746dd01849ec" containerName="revoke-keys"
Nov 03 21:04:09.360235 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.360209    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.441816 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.441783    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"]
Nov 03 21:04:09.465757 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.465715    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wk2w\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-kube-api-access-9wk2w\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.465988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.465842    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.465988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.465891    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.466188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.466163    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.466331 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.466311    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.466402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.466370    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.466468 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.466401    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.466468 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.466449    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.466569 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.466527    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f412fc-5b61-4234-adf0-d81571eb1b02-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568318 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568277    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568418    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568470    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568498    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568547    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568624    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f412fc-5b61-4234-adf0-d81571eb1b02-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568653    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wk2w\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-kube-api-access-9wk2w\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568709    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568743    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.568798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568761    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-credential-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.569200 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568869    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-certs\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.569200 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.568926    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-socket\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.569200 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.569146    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-data\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.569500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.569480    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f412fc-5b61-4234-adf0-d81571eb1b02-istiod-ca-cert\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.571183 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.571165    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-envoy\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.571392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.571375    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-podinfo\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.622540 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.622468    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-token\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.622771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.622745    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wk2w\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-kube-api-access-9wk2w\") pod \"e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") " pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.676202 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.676166    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:09.910339 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:09.910249    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:10.097117 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:10.086489    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:10.170762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:10.170697    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:04:10.170937 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:10.170806    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:04:10.170937 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:10.170850    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:04:10.175087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:10.175050    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"]
Nov 03 21:04:10.479794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:10.479755    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:10.524601 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:10.524568    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" event={"ID":"81f412fc-5b61-4234-adf0-d81571eb1b02","Type":"ContainerStarted","Data":"6af7628945615c6b4094ba0d43d471f02a2c0f88c03440639816ce813d43a1d8"}
Nov 03 21:04:10.640139 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:10.640062    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:11.562119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.558610    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" event={"ID":"81f412fc-5b61-4234-adf0-d81571eb1b02","Type":"ContainerStarted","Data":"ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448"}
Nov 03 21:04:11.676611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.676575    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:11.678736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.678696    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 21:04:11.678871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.678763    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 21:04:11.723155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.723075    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" podStartSLOduration=2.7230589739999997 podStartE2EDuration="2.723058974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:11.719641221 +0000 UTC m=+2091.723599965" watchObservedRunningTime="2026-11-03 21:04:11.723058974 +0000 UTC m=+2091.727017697"
Nov 03 21:04:11.922521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.922483    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"]
Nov 03 21:04:11.927398 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.927367    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:11.973517 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.973479    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:11.973816 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.973793    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:11.973931 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.973888    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2xh\" (UniqueName: \"kubernetes.io/projected/38f15403-bcd6-441c-ade1-a36c77d4c8f0-kube-api-access-xn2xh\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:11.976364 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.976338    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:11.976538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.976401    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:04:11.979361 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:11.979334    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"]
Nov 03 21:04:12.075455 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.075407    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2xh\" (UniqueName: \"kubernetes.io/projected/38f15403-bcd6-441c-ade1-a36c77d4c8f0-kube-api-access-xn2xh\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:12.075668 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.075558    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:12.075857 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.075838    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:12.076570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.076547    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:12.078307 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.078280    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:12.129374 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.129342    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2xh\" (UniqueName: \"kubernetes.io/projected/38f15403-bcd6-441c-ade1-a36c77d4c8f0-kube-api-access-xn2xh\") pod \"maas-api-revoke-sub-1c60b7dd-4wsll\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:12.270160 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.270034    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:12.674369 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.674328    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"]
Nov 03 21:04:12.677449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.677421    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 21:04:12.677582 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.677477    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 21:04:12.680105 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:04:12.680062    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f15403_bcd6_441c_ade1_a36c77d4c8f0.slice/crio-014a897967f3e5a3567f5b444ffe472bd6718abfc0b88c37049911ca6d2d4873 WatchSource:0}: Error finding container 014a897967f3e5a3567f5b444ffe472bd6718abfc0b88c37049911ca6d2d4873: Status 404 returned error can't find the container with id 014a897967f3e5a3567f5b444ffe472bd6718abfc0b88c37049911ca6d2d4873
Nov 03 21:04:12.756806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:12.756713    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:13.580674 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:13.580634    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll" event={"ID":"38f15403-bcd6-441c-ade1-a36c77d4c8f0","Type":"ContainerStarted","Data":"4d8f2536f8bedefe8bb97e24e038683557228cb4e03a8a448f9eea6604bad082"}
Nov 03 21:04:13.580861 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:13.580726    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll" event={"ID":"38f15403-bcd6-441c-ade1-a36c77d4c8f0","Type":"ContainerStarted","Data":"014a897967f3e5a3567f5b444ffe472bd6718abfc0b88c37049911ca6d2d4873"}
Nov 03 21:04:13.677161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:13.677074    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 21:04:13.677645 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:13.677197    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 21:04:13.704494 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:13.704438    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll" podStartSLOduration=2.7044249970000003 podStartE2EDuration="2.704424997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:13.70302431 +0000 UTC m=+2093.706983052" watchObservedRunningTime="2026-11-03 21:04:13.704424997 +0000 UTC m=+2093.708383722"
Nov 03 21:04:14.592720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:14.592686    5196 generic.go:356] "Generic (PLEG): container finished" podID="38f15403-bcd6-441c-ade1-a36c77d4c8f0" containerID="4d8f2536f8bedefe8bb97e24e038683557228cb4e03a8a448f9eea6604bad082" exitCode=0
Nov 03 21:04:14.592914 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:14.592806    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll" event={"ID":"38f15403-bcd6-441c-ade1-a36c77d4c8f0","Type":"ContainerDied","Data":"4d8f2536f8bedefe8bb97e24e038683557228cb4e03a8a448f9eea6604bad082"}
Nov 03 21:04:14.676903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:14.676870    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 21:04:14.677126 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:14.676926    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 21:04:15.062764 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.062660    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"]
Nov 03 21:04:15.067290 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.067256    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b5327d-sgx78"]
Nov 03 21:04:15.677538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.677492    5196 patch_prober.go:29] interesting pod/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 21:04:15.677716 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.677554    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 21:04:15.828413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.828385    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:15.980723 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.980626    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-openshift-ca\") pod \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") "
Nov 03 21:04:15.980723 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.980708    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f15403-bcd6-441c-ade1-a36c77d4c8f0-kube-api-access-xn2xh\" (UniqueName: \"kubernetes.io/projected/38f15403-bcd6-441c-ade1-a36c77d4c8f0-kube-api-access-xn2xh\") pod \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") "
Nov 03 21:04:15.980910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.980746    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-xks-ca\") pod \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\" (UID: \"38f15403-bcd6-441c-ade1-a36c77d4c8f0\") "
Nov 03 21:04:15.981522 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.981496    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-openshift-ca" pod "38f15403-bcd6-441c-ade1-a36c77d4c8f0" (UID: "38f15403-bcd6-441c-ade1-a36c77d4c8f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:15.982165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.982133    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.983624 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.983595    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-xks-ca" pod "38f15403-bcd6-441c-ade1-a36c77d4c8f0" (UID: "38f15403-bcd6-441c-ade1-a36c77d4c8f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:15.983970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:15.983943    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f15403-bcd6-441c-ade1-a36c77d4c8f0-kube-api-access-xn2xh" pod "38f15403-bcd6-441c-ade1-a36c77d4c8f0" (UID: "38f15403-bcd6-441c-ade1-a36c77d4c8f0"). InnerVolumeSpecName "kube-api-access-xn2xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:16.083864 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.083823    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2xh\" (UniqueName: \"kubernetes.io/projected/38f15403-bcd6-441c-ade1-a36c77d4c8f0-kube-api-access-xn2xh\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:16.083864 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.083857    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f15403-bcd6-441c-ade1-a36c77d4c8f0-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:16.114626 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.114586    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23504e58-91da-43d9-a79a-17b69fe84730" path="/var/lib/kubelet/pods/23504e58-91da-43d9-a79a-17b69fe84730/volumes"
Nov 03 21:04:16.204915 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.204876    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:16.420489 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.420450    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:16.612724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.612692    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"
Nov 03 21:04:16.612902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.612723    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll" event={"ID":"38f15403-bcd6-441c-ade1-a36c77d4c8f0","Type":"ContainerDied","Data":"014a897967f3e5a3567f5b444ffe472bd6718abfc0b88c37049911ca6d2d4873"}
Nov 03 21:04:16.612902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.612773    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="014a897967f3e5a3567f5b444ffe472bd6718abfc0b88c37049911ca6d2d4873"
Nov 03 21:04:16.741922 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.741836    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:16.742954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.742908    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:16.744860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:16.744839    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:04:18.488073 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:04:18.488035    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:04:18.496035 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:18.495999    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:20.518513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:20.518481    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:04:20.526223 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:20.526198    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:04:20.532809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:20.532782    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:04:20.540087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:20.540063    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:04:20.988810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:20.988779    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:21.144355 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:21.144320    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:23.930932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:23.930893    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:26.050508 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:26.050476    5196 scope.go:122] "RemoveContainer" containerID="6f879014f57fa9e157d700bbb384a13abe04aa8cf62b64a50b8e4d6493dd4843"
Nov 03 21:04:29.135402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:29.135362    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:34.185211 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:34.185169    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:34.375039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:34.374984    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:34.522913 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:34.522788    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:38.209232 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:38.209195    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:04:39.802802 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:39.802759    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:44.373772 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:44.373728    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:46.046412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:46.046346    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:46.316473 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:46.316381    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:46.478122 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:46.474228    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:46.656664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:46.656629    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"]
Nov 03 21:04:46.703912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:46.703876    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b6d800-wfntm"]
Nov 03 21:04:47.058793 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:47.058709    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:47.757981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:47.757941    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:48.111807 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:48.111771    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ce50c4-311e-4422-a31e-80136560a0ba" path="/var/lib/kubelet/pods/41ce50c4-311e-4422-a31e-80136560a0ba/volumes"
Nov 03 21:04:48.488490 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:04:48.488391    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:04:51.579279 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:51.579237    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:53.806599 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:53.806560    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:54.032441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:54.032403    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:54.201381 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:54.201332    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:54.985727 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:54.985672    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"]
Nov 03 21:04:54.986563 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:54.986544    5196 cpu_manager.go:414] "containerMap: removing container" podUID="38f15403-bcd6-441c-ade1-a36c77d4c8f0" containerName="revoke-keys"
Nov 03 21:04:54.986625 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:54.986566    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f15403-bcd6-441c-ade1-a36c77d4c8f0" containerName="revoke-keys"
Nov 03 21:04:54.986744 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:54.986734    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f15403-bcd6-441c-ade1-a36c77d4c8f0" containerName="revoke-keys"
Nov 03 21:04:54.991702 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:54.991683    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.044860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.044831    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"]
Nov 03 21:04:55.085188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.085138    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:55.085390 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.085274    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:04:55.099476 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.099443    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkfz\" (UniqueName: \"kubernetes.io/projected/474e51db-c707-45e8-99f9-7816d0fad649-kube-api-access-kfkfz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.099676 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.099555    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.099676 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.099585    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.201568 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.201532    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkfz\" (UniqueName: \"kubernetes.io/projected/474e51db-c707-45e8-99f9-7816d0fad649-kube-api-access-kfkfz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.201763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.201583    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.201763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.201711    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.202308 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.202286    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.204310 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.204287    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.256309 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.256225    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkfz\" (UniqueName: \"kubernetes.io/projected/474e51db-c707-45e8-99f9-7816d0fad649-kube-api-access-kfkfz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.308281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.308245    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:55.707872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:55.707833    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"]
Nov 03 21:04:55.713039 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:04:55.713008    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474e51db_c707_45e8_99f9_7816d0fad649.slice/crio-fe8ef29d318201bb6719ec3ae8b78eac7f1dee6094ef9f3157b9fc3673689177 WatchSource:0}: Error finding container fe8ef29d318201bb6719ec3ae8b78eac7f1dee6094ef9f3157b9fc3673689177: Status 404 returned error can't find the container with id fe8ef29d318201bb6719ec3ae8b78eac7f1dee6094ef9f3157b9fc3673689177
Nov 03 21:04:56.018801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:56.018722    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2" event={"ID":"474e51db-c707-45e8-99f9-7816d0fad649","Type":"ContainerStarted","Data":"fe8ef29d318201bb6719ec3ae8b78eac7f1dee6094ef9f3157b9fc3673689177"}
Nov 03 21:04:56.763469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:56.763402    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:57.028703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:57.028616    5196 generic.go:356] "Generic (PLEG): container finished" podID="474e51db-c707-45e8-99f9-7816d0fad649" containerID="7ce56764db8b39653c5f01bd333774ca62420afc85666b0c57405b5f40f70257" exitCode=0
Nov 03 21:04:57.028703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:57.028661    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2" event={"ID":"474e51db-c707-45e8-99f9-7816d0fad649","Type":"ContainerDied","Data":"7ce56764db8b39653c5f01bd333774ca62420afc85666b0c57405b5f40f70257"}
Nov 03 21:04:58.212334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.212305    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:58.260950 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.260916    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-xks-ca\") pod \"474e51db-c707-45e8-99f9-7816d0fad649\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") "
Nov 03 21:04:58.261186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.261044    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474e51db-c707-45e8-99f9-7816d0fad649-kube-api-access-kfkfz\" (UniqueName: \"kubernetes.io/projected/474e51db-c707-45e8-99f9-7816d0fad649-kube-api-access-kfkfz\") pod \"474e51db-c707-45e8-99f9-7816d0fad649\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") "
Nov 03 21:04:58.261186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.261117    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-openshift-ca\") pod \"474e51db-c707-45e8-99f9-7816d0fad649\" (UID: \"474e51db-c707-45e8-99f9-7816d0fad649\") "
Nov 03 21:04:58.261548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.261525    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-openshift-ca" pod "474e51db-c707-45e8-99f9-7816d0fad649" (UID: "474e51db-c707-45e8-99f9-7816d0fad649"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:58.261890 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.261870    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:58.263323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.263298    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-xks-ca" pod "474e51db-c707-45e8-99f9-7816d0fad649" (UID: "474e51db-c707-45e8-99f9-7816d0fad649"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:58.263522 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.263506    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474e51db-c707-45e8-99f9-7816d0fad649-kube-api-access-kfkfz" pod "474e51db-c707-45e8-99f9-7816d0fad649" (UID: "474e51db-c707-45e8-99f9-7816d0fad649"). InnerVolumeSpecName "kube-api-access-kfkfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:58.364299 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.364256    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474e51db-c707-45e8-99f9-7816d0fad649-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:58.364299 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:58.364296    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfkfz\" (UniqueName: \"kubernetes.io/projected/474e51db-c707-45e8-99f9-7816d0fad649-kube-api-access-kfkfz\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:59.046728 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:59.046694    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"
Nov 03 21:04:59.046728 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:59.046711    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2" event={"ID":"474e51db-c707-45e8-99f9-7816d0fad649","Type":"ContainerDied","Data":"fe8ef29d318201bb6719ec3ae8b78eac7f1dee6094ef9f3157b9fc3673689177"}
Nov 03 21:04:59.046954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:04:59.046756    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe8ef29d318201bb6719ec3ae8b78eac7f1dee6094ef9f3157b9fc3673689177"
Nov 03 21:05:01.804219 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:01.804180    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:06.538704 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:06.538663    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:06.970109 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:06.970053    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:07.186418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:07.177761    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:08.109320 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:08.109281    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:08.471188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:08.471078    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:08.636556 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:08.636512    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:08.763763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:08.763679    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"]
Nov 03 21:05:08.810306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:08.810269    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1ec1b6-44kvn"]
Nov 03 21:05:10.111988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:10.111950    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2f3547-8792-43a5-b67a-d10256e6dd1b" path="/var/lib/kubelet/pods/6b2f3547-8792-43a5-b67a-d10256e6dd1b/volumes"
Nov 03 21:05:12.924758 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:12.924719    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:14.488423 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:14.488379    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:14.704792 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:14.704748    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:14.862934 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:14.862896    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:15.026646 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:15.026606    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"]
Nov 03 21:05:15.073243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:15.073204    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e6aaa6-n4x7w"]
Nov 03 21:05:15.477010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:15.476978    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:16.111553 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:16.111522    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1a1a69-6464-4831-bae6-5f926c21585f" path="/var/lib/kubelet/pods/4a1a1a69-6464-4831-bae6-5f926c21585f/volumes"
Nov 03 21:05:18.487775 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:05:18.487734    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:05:19.207168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:19.207128    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:19.881668 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:19.881633    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:21.435728 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:21.435685    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:21.585222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:21.585183    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:21.761621 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:21.761534    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:24.524588 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:24.524503    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:25.218743 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:25.218704    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:25.378646 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:25.378604    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:25.599734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:25.599556    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:26.101914 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:26.101879    5196 scope.go:122] "RemoveContainer" containerID="55b31127592d8ddc3b99bd8047d9fa03ef4e1b5ca20e498a8dd7828f04d6b46c"
Nov 03 21:05:26.237912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:26.237874    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:26.506979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:26.506873    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:26.651720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:26.651684    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:26.765528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:26.765447    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"]
Nov 03 21:05:26.812456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:26.812422    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-80cbc0-rrjf2"]
Nov 03 21:05:27.151705 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.151674    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"]
Nov 03 21:05:27.152537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.152519    5196 cpu_manager.go:414] "containerMap: removing container" podUID="474e51db-c707-45e8-99f9-7816d0fad649" containerName="revoke-keys"
Nov 03 21:05:27.152537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.152537    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474e51db-c707-45e8-99f9-7816d0fad649" containerName="revoke-keys"
Nov 03 21:05:27.152663 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.152653    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="474e51db-c707-45e8-99f9-7816d0fad649" containerName="revoke-keys"
Nov 03 21:05:27.158335 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.158314    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.179913 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.179883    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:27.209997 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.209962    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"]
Nov 03 21:05:27.247222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.247178    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.247423 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.247274    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554sn\" (UniqueName: \"kubernetes.io/projected/21351d9d-6d91-4e0e-8a2f-b39db70ed597-kube-api-access-554sn\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.247513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.247448    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.250771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.250606    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:05:27.250771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.250696    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:05:27.306056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.304801    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"]
Nov 03 21:05:27.308681 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.308616    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy" containerID="cri-o://85163c644e145cf696074fcf96b6b654b0f236001799225c35473e389e52a7f7" gracePeriod=30
Nov 03 21:05:27.349530 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.349491    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554sn\" (UniqueName: \"kubernetes.io/projected/21351d9d-6d91-4e0e-8a2f-b39db70ed597-kube-api-access-554sn\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.349717 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.349670    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.349908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.349894    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.350593 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.350559    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.352321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.352300    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.415285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.415209    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554sn\" (UniqueName: \"kubernetes.io/projected/21351d9d-6d91-4e0e-8a2f-b39db70ed597-kube-api-access-554sn\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.477695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.477655    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:27.872058 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:27.872024    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"]
Nov 03 21:05:27.877599 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:05:27.877567    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21351d9d_6d91_4e0e_8a2f_b39db70ed597.slice/crio-d3e52c2839ccc3f3e72c182b70273f3986c01ce0e1bb9381a8cf09b6011c5ab3 WatchSource:0}: Error finding container d3e52c2839ccc3f3e72c182b70273f3986c01ce0e1bb9381a8cf09b6011c5ab3: Status 404 returned error can't find the container with id d3e52c2839ccc3f3e72c182b70273f3986c01ce0e1bb9381a8cf09b6011c5ab3
Nov 03 21:05:28.111240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:28.111208    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474e51db-c707-45e8-99f9-7816d0fad649" path="/var/lib/kubelet/pods/474e51db-c707-45e8-99f9-7816d0fad649/volumes"
Nov 03 21:05:28.563000 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:28.562904    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch" event={"ID":"21351d9d-6d91-4e0e-8a2f-b39db70ed597","Type":"ContainerStarted","Data":"769eff51ef1acbc17fbc6eb6f6fdcd24ae79cacfc0ad940168dc116d798b94c4"}
Nov 03 21:05:28.563000 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:28.562969    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch" event={"ID":"21351d9d-6d91-4e0e-8a2f-b39db70ed597","Type":"ContainerStarted","Data":"d3e52c2839ccc3f3e72c182b70273f3986c01ce0e1bb9381a8cf09b6011c5ab3"}
Nov 03 21:05:28.674170 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:28.674116    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch" podStartSLOduration=1.674086677 podStartE2EDuration="1.674086677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:28.670394979 +0000 UTC m=+2168.674353705" watchObservedRunningTime="2026-11-03 21:05:28.674086677 +0000 UTC m=+2168.678045400"
Nov 03 21:05:29.573495 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:29.573457    5196 generic.go:356] "Generic (PLEG): container finished" podID="21351d9d-6d91-4e0e-8a2f-b39db70ed597" containerID="769eff51ef1acbc17fbc6eb6f6fdcd24ae79cacfc0ad940168dc116d798b94c4" exitCode=0
Nov 03 21:05:29.573915 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:29.573546    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch" event={"ID":"21351d9d-6d91-4e0e-8a2f-b39db70ed597","Type":"ContainerDied","Data":"769eff51ef1acbc17fbc6eb6f6fdcd24ae79cacfc0ad940168dc116d798b94c4"}
Nov 03 21:05:30.813226 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.813194    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:30.819144 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.819069    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:30.906176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.906035    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-xks-ca\") pod \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") "
Nov 03 21:05:30.906176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.906107    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21351d9d-6d91-4e0e-8a2f-b39db70ed597-kube-api-access-554sn\" (UniqueName: \"kubernetes.io/projected/21351d9d-6d91-4e0e-8a2f-b39db70ed597-kube-api-access-554sn\") pod \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") "
Nov 03 21:05:30.906176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.906157    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-openshift-ca\") pod \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\" (UID: \"21351d9d-6d91-4e0e-8a2f-b39db70ed597\") "
Nov 03 21:05:30.906560 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.906534    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-openshift-ca" pod "21351d9d-6d91-4e0e-8a2f-b39db70ed597" (UID: "21351d9d-6d91-4e0e-8a2f-b39db70ed597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:30.906698 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.906682    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:30.908507 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.908480    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-xks-ca" pod "21351d9d-6d91-4e0e-8a2f-b39db70ed597" (UID: "21351d9d-6d91-4e0e-8a2f-b39db70ed597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:30.908646 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:30.908506    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21351d9d-6d91-4e0e-8a2f-b39db70ed597-kube-api-access-554sn" pod "21351d9d-6d91-4e0e-8a2f-b39db70ed597" (UID: "21351d9d-6d91-4e0e-8a2f-b39db70ed597"). InnerVolumeSpecName "kube-api-access-554sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:31.008540 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:31.008496    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21351d9d-6d91-4e0e-8a2f-b39db70ed597-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:31.008729 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:31.008569    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554sn\" (UniqueName: \"kubernetes.io/projected/21351d9d-6d91-4e0e-8a2f-b39db70ed597-kube-api-access-554sn\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:31.597747 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:31.597714    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"
Nov 03 21:05:31.597747 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:31.597728    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch" event={"ID":"21351d9d-6d91-4e0e-8a2f-b39db70ed597","Type":"ContainerDied","Data":"d3e52c2839ccc3f3e72c182b70273f3986c01ce0e1bb9381a8cf09b6011c5ab3"}
Nov 03 21:05:31.598016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:31.597787    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3e52c2839ccc3f3e72c182b70273f3986c01ce0e1bb9381a8cf09b6011c5ab3"
Nov 03 21:05:32.455784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.455743    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:32.609245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.609218    5196 generic.go:356] "Generic (PLEG): container finished" podID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerID="85163c644e145cf696074fcf96b6b654b0f236001799225c35473e389e52a7f7" exitCode=0
Nov 03 21:05:32.609401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.609285    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" event={"ID":"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb","Type":"ContainerDied","Data":"85163c644e145cf696074fcf96b6b654b0f236001799225c35473e389e52a7f7"}
Nov 03 21:05:32.609401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.609337    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws" event={"ID":"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb","Type":"ContainerDied","Data":"cbd22fa46a5ba7e8785cbc6fc086e4e29812bc015917f6859ab2a645c8ce72c6"}
Nov 03 21:05:32.609401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.609357    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbd22fa46a5ba7e8785cbc6fc086e4e29812bc015917f6859ab2a645c8ce72c6"
Nov 03 21:05:32.610972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.610947    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 21:05:32.717416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.717383    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:32.753655 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.753615    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-data\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.753872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.753733    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-socket\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.753872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.753781    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-token\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-token\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.753872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.753833    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-podinfo\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.754053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.753890    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-credential-socket\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.754053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.753889    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-data" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:32.754053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.753951    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-envoy\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.754053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.754033    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istiod-ca-cert\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.754330 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.754052    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-socket" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:32.754330 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.754068    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-kube-api-access-rhv77\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-kube-api-access-rhv77\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.754330 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.754298    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-certs\") pod \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\" (UID: \"7c7a8f53-7ecf-4333-b747-d8662ea2bdeb\") "
Nov 03 21:05:32.754671 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.754629    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-credential-socket" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:32.754671 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.754661    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istiod-ca-cert" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:32.754917 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.754821    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-certs" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:32.755835 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.755807    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.755975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.755840    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.755975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.755854    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.755975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.755868    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.755975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.755883    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.756612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.756584    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-podinfo" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:05:32.756750 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.756728    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-kube-api-access-rhv77" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "kube-api-access-rhv77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:32.757083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.757061    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-token" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:32.757199 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.757079    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-envoy" pod "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" (UID: "7c7a8f53-7ecf-4333-b747-d8662ea2bdeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:32.857976 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.857894    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.857976 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.857935    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.857976 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.857951    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.858216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.857982    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhv77\" (UniqueName: \"kubernetes.io/projected/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb-kube-api-access-rhv77\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.943117 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:32.925934    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:33.006198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.006156    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"]
Nov 03 21:05:33.034123 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.031196    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-20d356-6rlq9"]
Nov 03 21:05:33.072732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.072695    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:33.309433 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.309351    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:33.468122 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.467926    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:33.617996 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.617966    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"
Nov 03 21:05:33.822797 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.822760    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"]
Nov 03 21:05:33.869039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.868965    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-nzzws"]
Nov 03 21:05:33.977628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:33.977590    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:34.111391 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:34.111356    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" path="/var/lib/kubelet/pods/7c7a8f53-7ecf-4333-b747-d8662ea2bdeb/volumes"
Nov 03 21:05:34.111900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:34.111886    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938668d4-f1f1-4a19-8b4e-746dd01849ec" path="/var/lib/kubelet/pods/938668d4-f1f1-4a19-8b4e-746dd01849ec/volumes"
Nov 03 21:05:34.388769 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:34.388739    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:34.591388 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:34.591347    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:34.762041 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:34.761953    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:35.218119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:35.217651    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:35.611459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:35.611422    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:35.827719 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:35.826610    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:35.989974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:35.989877    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:36.144577 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:36.144540    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:37.494364 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:37.494320    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:38.145975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:38.145943    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:38.464342 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:38.464247    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:38.639178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:38.639070    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:38.853857 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:38.853815    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.104517 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.104414    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.278910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.278764    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.436944 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.436828    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.614759 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.614714    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.766605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.766475    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.913992 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.913958    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"]
Nov 03 21:05:39.914243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.914214    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy" containerID="cri-o://889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba" gracePeriod=30
Nov 03 21:05:39.927538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.927509    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"]
Nov 03 21:05:39.927782 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.927755    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy" containerID="cri-o://ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448" gracePeriod=30
Nov 03 21:05:39.931844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.931819    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"]
Nov 03 21:05:39.937021 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.936989    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.960745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:39.960704    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-j5mmv"]
Nov 03 21:05:40.152922 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:40.152873    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a255c4d-6f63-4166-afe3-7598df773d34" path="/var/lib/kubelet/pods/8a255c4d-6f63-4166-afe3-7598df773d34/volumes"
Nov 03 21:05:42.115502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:42.115463    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:42.662323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:42.662273    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:42.925436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:42.925153    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:43.083728 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.083693    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:43.117555 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.117526    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"]
Nov 03 21:05:43.118436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.118413    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy"
Nov 03 21:05:43.118436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.118436    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy"
Nov 03 21:05:43.118612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.118478    5196 cpu_manager.go:414] "containerMap: removing container" podUID="21351d9d-6d91-4e0e-8a2f-b39db70ed597" containerName="revoke-keys"
Nov 03 21:05:43.118612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.118484    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21351d9d-6d91-4e0e-8a2f-b39db70ed597" containerName="revoke-keys"
Nov 03 21:05:43.118612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.118611    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7a8f53-7ecf-4333-b747-d8662ea2bdeb" containerName="istio-proxy"
Nov 03 21:05:43.118749 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.118628    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="21351d9d-6d91-4e0e-8a2f-b39db70ed597" containerName="revoke-keys"
Nov 03 21:05:43.122559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.122541    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.290169 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.278606    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:05:43.339260 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.291127    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:05:43.377429 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.377395    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"]
Nov 03 21:05:43.379618 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.379577    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7p2j\" (UniqueName: \"kubernetes.io/projected/f2824fef-5afd-40af-ac3d-ec66583fcac1-kube-api-access-s7p2j\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.379901 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.379877    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.380054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.380033    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.481613 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.481567    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7p2j\" (UniqueName: \"kubernetes.io/projected/f2824fef-5afd-40af-ac3d-ec66583fcac1-kube-api-access-s7p2j\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.481801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.481742    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.481854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.481819    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.482572 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.482549    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.484601 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.484521    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.539958 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.538393    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7p2j\" (UniqueName: \"kubernetes.io/projected/f2824fef-5afd-40af-ac3d-ec66583fcac1-kube-api-access-s7p2j\") pod \"maas-api-revoke-sub-2a5e3e9c-lklbf\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:43.583217 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.583181    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:43.749183 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:43.749150    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:44.136636 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:44.136600    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"]
Nov 03 21:05:44.720912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:44.720873    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf" event={"ID":"f2824fef-5afd-40af-ac3d-ec66583fcac1","Type":"ContainerStarted","Data":"bed3abef20e5581cfe01c177ef74ebc89e661debbc0c39c2f36990a536bb9c52"}
Nov 03 21:05:44.721134 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:44.720940    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf" event={"ID":"f2824fef-5afd-40af-ac3d-ec66583fcac1","Type":"ContainerStarted","Data":"a0df24b62d9953c03bd1721c63a976b124bf0828efa54b2ccfb9be1b1e23a848"}
Nov 03 21:05:44.862573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:44.862523    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf" podStartSLOduration=1.862508138 podStartE2EDuration="1.862508138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:44.859038233 +0000 UTC m=+2184.862997013" watchObservedRunningTime="2026-11-03 21:05:44.862508138 +0000 UTC m=+2184.866466861"
Nov 03 21:05:45.028398 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.028344    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:45.308741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.308710    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:05:45.395774 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.395752    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:05:45.429138 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429061    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-socket\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.429336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429212    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-data\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.429336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429296    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-podinfo\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.429336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429327    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-envoy\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.429500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429378    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-socket" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.429548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429517    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-data" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.429548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429523    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-kube-api-access-v66w2\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-kube-api-access-v66w2\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.429643 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429626    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-certs\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.429699 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429663    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-credential-socket\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.429755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429738    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-token\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.429834 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.429816    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istiod-ca-cert\") pod \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\" (UID: \"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c\") "
Nov 03 21:05:45.430336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.430223    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-certs" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.430336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.430290    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-credential-socket" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.430746 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.430615    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istiod-ca-cert" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:45.430746 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.430631    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.430746 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.430656    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.430746 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.430669    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.430746 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.430682    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.432384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.432352    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-token" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:45.432483 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.432424    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-envoy" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.432483 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.432451    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-kube-api-access-v66w2" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "kube-api-access-v66w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:45.432750 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.432733    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-podinfo" pod "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" (UID: "8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:05:45.532206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532163    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-socket\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.532521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532234    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-data\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.532521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532277    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-token\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-token\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.532521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532311    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-envoy\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.532521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532463    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f412fc-5b61-4234-adf0-d81571eb1b02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f412fc-5b61-4234-adf0-d81571eb1b02-istiod-ca-cert\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.532521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532493    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-kube-api-access-9wk2w\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-kube-api-access-9wk2w\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.532521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532505    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-socket" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.532922 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532526    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-credential-socket\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.532922 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532739    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-podinfo\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.532922 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.532793    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-certs\") pod \"81f412fc-5b61-4234-adf0-d81571eb1b02\" (UID: \"81f412fc-5b61-4234-adf0-d81571eb1b02\") "
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533373    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-data" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533633    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-certs" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533643    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533662    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533679    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533692    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533706    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v66w2\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-kube-api-access-v66w2\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533722    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533736    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.533963    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-credential-socket" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.534516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.534483    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f412fc-5b61-4234-adf0-d81571eb1b02-istiod-ca-cert" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:45.536083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.536046    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-envoy" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:45.536083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.536063    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-token" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:45.536362 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.536339    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-kube-api-access-9wk2w" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "kube-api-access-9wk2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:45.538339 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.538313    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-podinfo" pod "81f412fc-5b61-4234-adf0-d81571eb1b02" (UID: "81f412fc-5b61-4234-adf0-d81571eb1b02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:05:45.635046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.635007    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f412fc-5b61-4234-adf0-d81571eb1b02-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.635046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.635040    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wk2w\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-kube-api-access-9wk2w\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.635046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.635052    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.635338 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.635061    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.635338 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.635070    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.635338 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.635080    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.635338 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.635088    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f412fc-5b61-4234-adf0-d81571eb1b02-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:45.734564 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.734464    5196 generic.go:356] "Generic (PLEG): container finished" podID="f2824fef-5afd-40af-ac3d-ec66583fcac1" containerID="bed3abef20e5581cfe01c177ef74ebc89e661debbc0c39c2f36990a536bb9c52" exitCode=0
Nov 03 21:05:45.734753 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.734619    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf" event={"ID":"f2824fef-5afd-40af-ac3d-ec66583fcac1","Type":"ContainerDied","Data":"bed3abef20e5581cfe01c177ef74ebc89e661debbc0c39c2f36990a536bb9c52"}
Nov 03 21:05:45.737277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.737241    5196 generic.go:356] "Generic (PLEG): container finished" podID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerID="889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba" exitCode=0
Nov 03 21:05:45.737435 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.737302    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"
Nov 03 21:05:45.737435 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.737318    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" event={"ID":"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c","Type":"ContainerDied","Data":"889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba"}
Nov 03 21:05:45.737435 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.737383    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4" event={"ID":"8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c","Type":"ContainerDied","Data":"34e7057308a4fab42e69bdfaa33866c00fcef8c5e3895638548bcf2ff8cc09a8"}
Nov 03 21:05:45.737435 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.737413    5196 scope.go:122] "RemoveContainer" containerID="889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba"
Nov 03 21:05:45.739818 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.739780    5196 generic.go:356] "Generic (PLEG): container finished" podID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerID="ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448" exitCode=0
Nov 03 21:05:45.739977 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.739959    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" event={"ID":"81f412fc-5b61-4234-adf0-d81571eb1b02","Type":"ContainerDied","Data":"ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448"}
Nov 03 21:05:45.740042 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.740013    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44" event={"ID":"81f412fc-5b61-4234-adf0-d81571eb1b02","Type":"ContainerDied","Data":"6af7628945615c6b4094ba0d43d471f02a2c0f88c03440639816ce813d43a1d8"}
Nov 03 21:05:45.740145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.740129    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"
Nov 03 21:05:45.752763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.752741    5196 scope.go:122] "RemoveContainer" containerID="889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba"
Nov 03 21:05:45.753204 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:05:45.753173    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba\": container with ID starting with 889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba not found: ID does not exist" containerID="889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba"
Nov 03 21:05:45.753300 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.753212    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba"} err="failed to get container status \"889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba\": rpc error: code = NotFound desc = could not find container \"889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba\": container with ID starting with 889bdd96385a20290d3013c5927edb04fb463f6ed96ff8d806cf1ffe005b21ba not found: ID does not exist"
Nov 03 21:05:45.753300 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.753239    5196 scope.go:122] "RemoveContainer" containerID="ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448"
Nov 03 21:05:45.774605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.774574    5196 scope.go:122] "RemoveContainer" containerID="ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448"
Nov 03 21:05:45.775002 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:05:45.774980    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448\": container with ID starting with ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448 not found: ID does not exist" containerID="ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448"
Nov 03 21:05:45.775525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.775007    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448"} err="failed to get container status \"ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448\": rpc error: code = NotFound desc = could not find container \"ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448\": container with ID starting with ecca6d8ef791109ff889dbbdc60ef57ed78aafe6366289276c69a755ecaf8448 not found: ID does not exist"
Nov 03 21:05:45.961576 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.961511    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"]
Nov 03 21:05:45.966025 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:45.965991    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-g6x44"]
Nov 03 21:05:46.068251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:46.068222    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"]
Nov 03 21:05:46.073039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:46.073010    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-20d356-openshift-default-7f6c56495f-m5mr4"]
Nov 03 21:05:46.112994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:46.112961    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" path="/var/lib/kubelet/pods/81f412fc-5b61-4234-adf0-d81571eb1b02/volumes"
Nov 03 21:05:46.113532 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:46.113517    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" path="/var/lib/kubelet/pods/8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c/volumes"
Nov 03 21:05:46.928311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:46.928286    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:47.059763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.059669    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-openshift-ca\") pod \"f2824fef-5afd-40af-ac3d-ec66583fcac1\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") "
Nov 03 21:05:47.059951 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.059875    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2824fef-5afd-40af-ac3d-ec66583fcac1-kube-api-access-s7p2j\" (UniqueName: \"kubernetes.io/projected/f2824fef-5afd-40af-ac3d-ec66583fcac1-kube-api-access-s7p2j\") pod \"f2824fef-5afd-40af-ac3d-ec66583fcac1\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") "
Nov 03 21:05:47.060030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.059988    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-xks-ca\") pod \"f2824fef-5afd-40af-ac3d-ec66583fcac1\" (UID: \"f2824fef-5afd-40af-ac3d-ec66583fcac1\") "
Nov 03 21:05:47.060030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.060016    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-openshift-ca" pod "f2824fef-5afd-40af-ac3d-ec66583fcac1" (UID: "f2824fef-5afd-40af-ac3d-ec66583fcac1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:47.060717 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.060688    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:47.062380 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.062348    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2824fef-5afd-40af-ac3d-ec66583fcac1-kube-api-access-s7p2j" pod "f2824fef-5afd-40af-ac3d-ec66583fcac1" (UID: "f2824fef-5afd-40af-ac3d-ec66583fcac1"). InnerVolumeSpecName "kube-api-access-s7p2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:47.062800 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.062764    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-xks-ca" pod "f2824fef-5afd-40af-ac3d-ec66583fcac1" (UID: "f2824fef-5afd-40af-ac3d-ec66583fcac1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:47.161730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.161680    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2824fef-5afd-40af-ac3d-ec66583fcac1-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:47.161730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.161727    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7p2j\" (UniqueName: \"kubernetes.io/projected/f2824fef-5afd-40af-ac3d-ec66583fcac1-kube-api-access-s7p2j\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:47.762929 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.762874    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf" event={"ID":"f2824fef-5afd-40af-ac3d-ec66583fcac1","Type":"ContainerDied","Data":"a0df24b62d9953c03bd1721c63a976b124bf0828efa54b2ccfb9be1b1e23a848"}
Nov 03 21:05:47.763162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.762967    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0df24b62d9953c03bd1721c63a976b124bf0828efa54b2ccfb9be1b1e23a848"
Nov 03 21:05:47.763162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.762891    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"
Nov 03 21:05:47.838532 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:47.838489    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:48.093965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:48.093930    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:48.248665 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:48.248634    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:48.416979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:48.416889    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:48.487687 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:05:48.487649    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:05:50.277336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:50.277290    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:51.000259 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:51.000226    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:05:51.105274 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:51.105221    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:51.341380 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:51.337914    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:53.884239 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:53.884203    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:55.278129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:55.278076    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:55.934960 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:55.934924    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:59.532648 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:59.532614    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:59.695306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:59.695268    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:59.845731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:05:59.845693    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:00.152393 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:00.152293    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:01.864839 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:01.864800    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:02.053552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:02.053510    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:05.529411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.529375    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:05.969856 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.969812    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"]
Nov 03 21:06:05.971723 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971698    5196 cpu_manager.go:414] "containerMap: removing container" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy"
Nov 03 21:06:05.971723 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971720    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy"
Nov 03 21:06:05.971902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971759    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f2824fef-5afd-40af-ac3d-ec66583fcac1" containerName="revoke-keys"
Nov 03 21:06:05.971902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971764    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2824fef-5afd-40af-ac3d-ec66583fcac1" containerName="revoke-keys"
Nov 03 21:06:05.971902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971774    5196 cpu_manager.go:414] "containerMap: removing container" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy"
Nov 03 21:06:05.971902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971779    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy"
Nov 03 21:06:05.972024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971923    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2824fef-5afd-40af-ac3d-ec66583fcac1" containerName="revoke-keys"
Nov 03 21:06:05.972024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971933    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8d15ba-871f-4bd4-bbbb-a2f2d40ae93c" containerName="istio-proxy"
Nov 03 21:06:05.972024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.971946    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f412fc-5b61-4234-adf0-d81571eb1b02" containerName="istio-proxy"
Nov 03 21:06:05.977088 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:05.977069    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.022486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.022452    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"]
Nov 03 21:06:06.028625 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.028590    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:06:06.075786 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.075756    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:06:06.132441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.132406    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.132631 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.132546    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.132692 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.132643    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkpvx\" (UniqueName: \"kubernetes.io/projected/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-kube-api-access-pkpvx\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.234655 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.234568    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.234655 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.234637    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkpvx\" (UniqueName: \"kubernetes.io/projected/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-kube-api-access-pkpvx\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.234893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.234863    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.235515 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.235493    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.237203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.237177    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.289394 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.289360    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkpvx\" (UniqueName: \"kubernetes.io/projected/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-kube-api-access-pkpvx\") pod \"maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.292075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.292055    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:06.673378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.673340    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"]
Nov 03 21:06:06.679202 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:06:06.679164    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f3bdbb_a243_47f8_ad08_de63e4ccf5ec.slice/crio-6bd4cb2375ed7f7ef7a7978e0a725d2992ce3b578e735a5b02a34907e848ac55 WatchSource:0}: Error finding container 6bd4cb2375ed7f7ef7a7978e0a725d2992ce3b578e735a5b02a34907e848ac55: Status 404 returned error can't find the container with id 6bd4cb2375ed7f7ef7a7978e0a725d2992ce3b578e735a5b02a34907e848ac55
Nov 03 21:06:06.981961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:06.981865    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m" event={"ID":"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec","Type":"ContainerStarted","Data":"6bd4cb2375ed7f7ef7a7978e0a725d2992ce3b578e735a5b02a34907e848ac55"}
Nov 03 21:06:07.517657 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:07.517624    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:07.659491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:07.659462    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:07.993295 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:07.993255    5196 generic.go:356] "Generic (PLEG): container finished" podID="22f3bdbb-a243-47f8-ad08-de63e4ccf5ec" containerID="f573ecb6e16a453e53ebd46b6eb5e561961c56012624b483eebc3450714a3b26" exitCode=0
Nov 03 21:06:07.993756 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:07.993376    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m" event={"ID":"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec","Type":"ContainerDied","Data":"f573ecb6e16a453e53ebd46b6eb5e561961c56012624b483eebc3450714a3b26"}
Nov 03 21:06:08.515836 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:08.515800    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:09.173840 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.173814    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:09.280846 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.280746    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-kube-api-access-pkpvx\" (UniqueName: \"kubernetes.io/projected/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-kube-api-access-pkpvx\") pod \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") "
Nov 03 21:06:09.281378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.281343    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-xks-ca\") pod \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") "
Nov 03 21:06:09.281528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.281471    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-openshift-ca\") pod \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\" (UID: \"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec\") "
Nov 03 21:06:09.281828 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.281799    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-openshift-ca" pod "22f3bdbb-a243-47f8-ad08-de63e4ccf5ec" (UID: "22f3bdbb-a243-47f8-ad08-de63e4ccf5ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:09.282311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.282288    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:09.284065 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.284036    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-xks-ca" pod "22f3bdbb-a243-47f8-ad08-de63e4ccf5ec" (UID: "22f3bdbb-a243-47f8-ad08-de63e4ccf5ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:09.284227 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.284132    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-kube-api-access-pkpvx" pod "22f3bdbb-a243-47f8-ad08-de63e4ccf5ec" (UID: "22f3bdbb-a243-47f8-ad08-de63e4ccf5ec"). InnerVolumeSpecName "kube-api-access-pkpvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:09.383890 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.383851    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:09.383890 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:09.383887    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkpvx\" (UniqueName: \"kubernetes.io/projected/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec-kube-api-access-pkpvx\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:10.013395 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.013366    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"
Nov 03 21:06:10.013395 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.013373    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m" event={"ID":"22f3bdbb-a243-47f8-ad08-de63e4ccf5ec","Type":"ContainerDied","Data":"6bd4cb2375ed7f7ef7a7978e0a725d2992ce3b578e735a5b02a34907e848ac55"}
Nov 03 21:06:10.013625 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.013451    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd4cb2375ed7f7ef7a7978e0a725d2992ce3b578e735a5b02a34907e848ac55"
Nov 03 21:06:10.213799 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.213766    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"]
Nov 03 21:06:10.214690 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.214669    5196 cpu_manager.go:414] "containerMap: removing container" podUID="22f3bdbb-a243-47f8-ad08-de63e4ccf5ec" containerName="revoke-keys"
Nov 03 21:06:10.214690 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.214692    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f3bdbb-a243-47f8-ad08-de63e4ccf5ec" containerName="revoke-keys"
Nov 03 21:06:10.214826 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.214817    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f3bdbb-a243-47f8-ad08-de63e4ccf5ec" containerName="revoke-keys"
Nov 03 21:06:10.218799 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.218778    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.270939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.270843    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"]
Nov 03 21:06:10.298747 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.298709    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnh9m\" (UniqueName: \"kubernetes.io/projected/298784fb-3289-4967-86cf-1b69c65d6a68-kube-api-access-qnh9m\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.298979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.298954    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.299054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.298998    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.312228 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.312197    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:06:10.314774 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.314746    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:06:10.400898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.400854    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnh9m\" (UniqueName: \"kubernetes.io/projected/298784fb-3289-4967-86cf-1b69c65d6a68-kube-api-access-qnh9m\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.401139 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.401042    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.401139 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.401079    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.401812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.401788    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.403775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.403752    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.454444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.454417    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnh9m\" (UniqueName: \"kubernetes.io/projected/298784fb-3289-4967-86cf-1b69c65d6a68-kube-api-access-qnh9m\") pod \"maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.536818 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.536732    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:10.778756 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.778717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:10.960370 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:10.960331    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"]
Nov 03 21:06:10.966024 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:06:10.965975    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298784fb_3289_4967_86cf_1b69c65d6a68.slice/crio-8ec377cefc08eaeeb8e222387a2dd767abf5f6f412735843d540886c8cd2a418 WatchSource:0}: Error finding container 8ec377cefc08eaeeb8e222387a2dd767abf5f6f412735843d540886c8cd2a418: Status 404 returned error can't find the container with id 8ec377cefc08eaeeb8e222387a2dd767abf5f6f412735843d540886c8cd2a418
Nov 03 21:06:11.023955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:11.023921    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4" event={"ID":"298784fb-3289-4967-86cf-1b69c65d6a68","Type":"ContainerStarted","Data":"8ec377cefc08eaeeb8e222387a2dd767abf5f6f412735843d540886c8cd2a418"}
Nov 03 21:06:12.033976 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.033936    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4" event={"ID":"298784fb-3289-4967-86cf-1b69c65d6a68","Type":"ContainerStarted","Data":"3e28c93829dfe8c69ea66225bb5823cc657586add841eab99cc452e4abf68cca"}
Nov 03 21:06:12.187946 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.187881    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4" podStartSLOduration=2.18786767 podStartE2EDuration="2.18786767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:12.14039454 +0000 UTC m=+2212.144353301" watchObservedRunningTime="2026-11-03 21:06:12.18786767 +0000 UTC m=+2212.191826395"
Nov 03 21:06:12.649351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.649313    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"]
Nov 03 21:06:12.654322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.654297    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.747985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.747951    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.748238 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.748025    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.748238 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.748196    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqccg\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-kube-api-access-wqccg\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.748398 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.748273    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.748521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.748489    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.748575 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.748542    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.748653 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.748633    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.748710 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.748677    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.748779 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.748766    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.850928 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.850887    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.851166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.850942    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.851166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.850996    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.851166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.851034    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.851166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.851081    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.851474 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.851201    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.851474 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.851272    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.851474 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.851338    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqccg\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-kube-api-access-wqccg\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.851474 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.851414    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.855356 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.855305    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.855626 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.855590    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.861113 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.856742    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.866792 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.866750    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"]
Nov 03 21:06:12.871163 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.871127    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.878378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.878346    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.878705 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.878681    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.881372 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.881326    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.909772 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.909695    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqccg\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-kube-api-access-wqccg\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.910000 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.909963    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qnkkw\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:12.969604 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:12.969568    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:13.049578 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:13.049535    5196 generic.go:356] "Generic (PLEG): container finished" podID="298784fb-3289-4967-86cf-1b69c65d6a68" containerID="3e28c93829dfe8c69ea66225bb5823cc657586add841eab99cc452e4abf68cca" exitCode=0
Nov 03 21:06:13.050021 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:13.049683    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4" event={"ID":"298784fb-3289-4967-86cf-1b69c65d6a68","Type":"ContainerDied","Data":"3e28c93829dfe8c69ea66225bb5823cc657586add841eab99cc452e4abf68cca"}
Nov 03 21:06:13.408995 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:13.408962    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"]
Nov 03 21:06:13.414246 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:06:13.414219    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a05f2b3_49f5_4ee3_b07b_555e4da13d13.slice/crio-135ce48e7745e8285102f9f267aeddccb64e76b34ee3ec5e89df96c60fbdcfcd WatchSource:0}: Error finding container 135ce48e7745e8285102f9f267aeddccb64e76b34ee3ec5e89df96c60fbdcfcd: Status 404 returned error can't find the container with id 135ce48e7745e8285102f9f267aeddccb64e76b34ee3ec5e89df96c60fbdcfcd
Nov 03 21:06:13.417212 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:13.417177    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:13.417322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:13.417247    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:13.417322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:13.417286    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:14.072676 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.072637    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" event={"ID":"9a05f2b3-49f5-4ee3-b07b-555e4da13d13","Type":"ContainerStarted","Data":"bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b"}
Nov 03 21:06:14.073304 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.072706    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" event={"ID":"9a05f2b3-49f5-4ee3-b07b-555e4da13d13","Type":"ContainerStarted","Data":"135ce48e7745e8285102f9f267aeddccb64e76b34ee3ec5e89df96c60fbdcfcd"}
Nov 03 21:06:14.283890 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.283838    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podStartSLOduration=2.283823465 podStartE2EDuration="2.283823465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:14.233897751 +0000 UTC m=+2214.237856488" watchObservedRunningTime="2026-11-03 21:06:14.283823465 +0000 UTC m=+2214.287782188"
Nov 03 21:06:14.307279 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.307225    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:14.377565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.377457    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298784fb-3289-4967-86cf-1b69c65d6a68-kube-api-access-qnh9m\" (UniqueName: \"kubernetes.io/projected/298784fb-3289-4967-86cf-1b69c65d6a68-kube-api-access-qnh9m\") pod \"298784fb-3289-4967-86cf-1b69c65d6a68\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") "
Nov 03 21:06:14.377830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.377685    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-xks-ca\") pod \"298784fb-3289-4967-86cf-1b69c65d6a68\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") "
Nov 03 21:06:14.377921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.377877    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-openshift-ca\") pod \"298784fb-3289-4967-86cf-1b69c65d6a68\" (UID: \"298784fb-3289-4967-86cf-1b69c65d6a68\") "
Nov 03 21:06:14.378259 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.378232    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-openshift-ca" pod "298784fb-3289-4967-86cf-1b69c65d6a68" (UID: "298784fb-3289-4967-86cf-1b69c65d6a68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:14.378610 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.378589    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:14.379875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.379846    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298784fb-3289-4967-86cf-1b69c65d6a68-kube-api-access-qnh9m" pod "298784fb-3289-4967-86cf-1b69c65d6a68" (UID: "298784fb-3289-4967-86cf-1b69c65d6a68"). InnerVolumeSpecName "kube-api-access-qnh9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:14.379985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.379896    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-xks-ca" pod "298784fb-3289-4967-86cf-1b69c65d6a68" (UID: "298784fb-3289-4967-86cf-1b69c65d6a68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:14.480159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.480126    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298784fb-3289-4967-86cf-1b69c65d6a68-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:14.480159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.480158    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnh9m\" (UniqueName: \"kubernetes.io/projected/298784fb-3289-4967-86cf-1b69c65d6a68-kube-api-access-qnh9m\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:14.970791 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.970748    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:14.972714 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.972682    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 21:06:14.972863 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:14.972740    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 21:06:15.098578 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:15.098191    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"
Nov 03 21:06:15.098578 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:15.098185    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4" event={"ID":"298784fb-3289-4967-86cf-1b69c65d6a68","Type":"ContainerDied","Data":"8ec377cefc08eaeeb8e222387a2dd767abf5f6f412735843d540886c8cd2a418"}
Nov 03 21:06:15.099691 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:15.099666    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ec377cefc08eaeeb8e222387a2dd767abf5f6f412735843d540886c8cd2a418"
Nov 03 21:06:15.870780 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:15.870743    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:15.970676 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:15.970634    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 21:06:15.970899 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:15.970752    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 21:06:15.986497 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:15.986282    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"]
Nov 03 21:06:15.987122 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:15.986739    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy" containerID="cri-o://47808c30d8a8e4aecc3db4ba394dbf7e335d3733e56fd5418bd6c122a563aee0" gracePeriod=30
Nov 03 21:06:16.970671 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:16.970632    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 21:06:16.971088 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:16.970720    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 21:06:17.970229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:17.970180    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 21:06:17.970430 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:17.970249    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 21:06:18.364655 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:18.364612    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"]
Nov 03 21:06:18.365402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:18.365368    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy" containerID="cri-o://30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20" gracePeriod=30
Nov 03 21:06:18.506932 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:06:18.506892    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:06:18.970393 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:18.970359    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 21:06:18.970578 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:18.970449    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 21:06:19.360218 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:19.360180    5196 patch_prober.go:29] interesting pod/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:06:19.360436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:19.360358    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:06:19.811444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:19.811339    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:19.949982 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:19.949941    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:06:19.950215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:19.950017    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:06:19.970964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:19.970919    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 21:06:19.971238 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:19.970995    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 21:06:20.073504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:20.073442    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:20.941116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:20.941037    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"]
Nov 03 21:06:20.941961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:20.941924    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf" podUID="103c0f10-3b8e-4747-b25a-84634edc6a3a" containerName="discovery" containerID="cri-o://672c73336600261964330cec63d1713ce59f54dba584c0e0067eea26cc20e814" gracePeriod=30
Nov 03 21:06:21.051514 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.040371    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Nov 03 21:06:21.051514 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.040448    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Nov 03 21:06:21.107715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.107620    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:21.224961 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:06:21.211362    5196 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f17ae4d_acf3_4475_b7d2_1c9e8704a0de.slice/crio-conmon-47808c30d8a8e4aecc3db4ba394dbf7e335d3733e56fd5418bd6c122a563aee0.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 21:06:21.377329 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.377285    5196 generic.go:356] "Generic (PLEG): container finished" podID="103c0f10-3b8e-4747-b25a-84634edc6a3a" containerID="672c73336600261964330cec63d1713ce59f54dba584c0e0067eea26cc20e814" exitCode=0
Nov 03 21:06:21.377543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.377526    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf" event={"ID":"103c0f10-3b8e-4747-b25a-84634edc6a3a","Type":"ContainerDied","Data":"672c73336600261964330cec63d1713ce59f54dba584c0e0067eea26cc20e814"}
Nov 03 21:06:21.379488 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.379464    5196 generic.go:356] "Generic (PLEG): container finished" podID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerID="47808c30d8a8e4aecc3db4ba394dbf7e335d3733e56fd5418bd6c122a563aee0" exitCode=0
Nov 03 21:06:21.379610 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.379538    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" event={"ID":"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de","Type":"ContainerDied","Data":"47808c30d8a8e4aecc3db4ba394dbf7e335d3733e56fd5418bd6c122a563aee0"}
Nov 03 21:06:21.454778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.454745    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 21:06:21.503407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.503382    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 21:06:21.507795 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.507768    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-certs\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.507912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.507830    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-ca-configmap\") pod \"103c0f10-3b8e-4747-b25a-84634edc6a3a\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") "
Nov 03 21:06:21.507912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.507875    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-envoy\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.508028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508010    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-certs" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:21.508129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508086    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103c0f10-3b8e-4747-b25a-84634edc6a3a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/103c0f10-3b8e-4747-b25a-84634edc6a3a-local-certs\") pod \"103c0f10-3b8e-4747-b25a-84634edc6a3a\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") "
Nov 03 21:06:21.508208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508179    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-kubeconfig\") pod \"103c0f10-3b8e-4747-b25a-84634edc6a3a\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") "
Nov 03 21:06:21.508279 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508263    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-podinfo\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.508343 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508311    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-credential-socket\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.508401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508343    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-token\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-token\") pod \"103c0f10-3b8e-4747-b25a-84634edc6a3a\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") "
Nov 03 21:06:21.508401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508380    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-socket\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.508516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508428    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-dns-cert\") pod \"103c0f10-3b8e-4747-b25a-84634edc6a3a\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") "
Nov 03 21:06:21.508516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508501    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-cacerts\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-cacerts\") pod \"103c0f10-3b8e-4747-b25a-84634edc6a3a\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") "
Nov 03 21:06:21.508644 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508544    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istiod-ca-cert\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.508644 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508597    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-kube-api-access-2248m\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-kube-api-access-2248m\") pod \"103c0f10-3b8e-4747-b25a-84634edc6a3a\" (UID: \"103c0f10-3b8e-4747-b25a-84634edc6a3a\") "
Nov 03 21:06:21.508741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.508681    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-token\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-token\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.509391 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.509356    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-ca-configmap" pod "103c0f10-3b8e-4747-b25a-84634edc6a3a" (UID: "103c0f10-3b8e-4747-b25a-84634edc6a3a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:21.509391 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.509381    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-credential-socket" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:21.509572 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.509398    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-socket" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:21.509572 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.509426    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-kube-api-access-bz2zg\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-kube-api-access-bz2zg\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.509572 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.509498    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-data\") pod \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\" (UID: \"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de\") "
Nov 03 21:06:21.509572 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.509532    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istiod-ca-cert" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:21.510376 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.510350    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-data" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:21.510887 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.510864    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.510981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.510894    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.510981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.510910    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.510981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.510923    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.510981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.510936    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.510981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.510950    5196 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-ca-configmap\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.513966 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.513941    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-token" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:21.513966 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.513948    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-podinfo" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:06:21.514186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.514063    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-kube-api-access-2248m" pod "103c0f10-3b8e-4747-b25a-84634edc6a3a" (UID: "103c0f10-3b8e-4747-b25a-84634edc6a3a"). InnerVolumeSpecName "kube-api-access-2248m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:21.514186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.514129    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-kubeconfig" pod "103c0f10-3b8e-4747-b25a-84634edc6a3a" (UID: "103c0f10-3b8e-4747-b25a-84634edc6a3a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:21.514186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.514168    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-token" pod "103c0f10-3b8e-4747-b25a-84634edc6a3a" (UID: "103c0f10-3b8e-4747-b25a-84634edc6a3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:21.514538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.514228    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-envoy" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:21.514538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.514258    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103c0f10-3b8e-4747-b25a-84634edc6a3a-local-certs" pod "103c0f10-3b8e-4747-b25a-84634edc6a3a" (UID: "103c0f10-3b8e-4747-b25a-84634edc6a3a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:21.514538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.514331    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-cacerts" pod "103c0f10-3b8e-4747-b25a-84634edc6a3a" (UID: "103c0f10-3b8e-4747-b25a-84634edc6a3a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:21.515444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.515415    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-dns-cert" pod "103c0f10-3b8e-4747-b25a-84634edc6a3a" (UID: "103c0f10-3b8e-4747-b25a-84634edc6a3a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:21.515870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.515848    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-kube-api-access-bz2zg" pod "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" (UID: "7f17ae4d-acf3-4475-b7d2-1c9e8704a0de"). InnerVolumeSpecName "kube-api-access-bz2zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:21.613277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613230    5196 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-csr-dns-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613300    5196 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-cacerts\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613320    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2248m\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-kube-api-access-2248m\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613348    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613363    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2zg\" (UniqueName: \"kubernetes.io/projected/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-kube-api-access-bz2zg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613379    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613394    5196 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/103c0f10-3b8e-4747-b25a-84634edc6a3a-local-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613407    5196 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-kubeconfig\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613419    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:21.613460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:21.613433    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103c0f10-3b8e-4747-b25a-84634edc6a3a-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:22.310739 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.310637    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:22.408950 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.408918    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"
Nov 03 21:06:22.411683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.411602    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq" event={"ID":"7f17ae4d-acf3-4475-b7d2-1c9e8704a0de","Type":"ContainerDied","Data":"af2de49713e91fe0e2fc9555e6307c4b545a8da01d6ec1b402a7e758e7a9d77e"}
Nov 03 21:06:22.411907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.411736    5196 scope.go:122] "RemoveContainer" containerID="47808c30d8a8e4aecc3db4ba394dbf7e335d3733e56fd5418bd6c122a563aee0"
Nov 03 21:06:22.415078 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.415045    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf" event={"ID":"103c0f10-3b8e-4747-b25a-84634edc6a3a","Type":"ContainerDied","Data":"1c31be20b767acc8bfb5126365dab8b857fdb1714c145bfbdbca3a0a4387a880"}
Nov 03 21:06:22.415280 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.415203    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"
Nov 03 21:06:22.429533 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.428264    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:22.430296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.429754    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:22.432045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.432000    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:06:22.438211 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.438155    5196 scope.go:122] "RemoveContainer" containerID="672c73336600261964330cec63d1713ce59f54dba584c0e0067eea26cc20e814"
Nov 03 21:06:22.512490 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.512456    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:22.572324 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.572290    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"]
Nov 03 21:06:22.618445 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.618410    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4nvbf"]
Nov 03 21:06:22.722541 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.722507    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"]
Nov 03 21:06:22.769288 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:22.769246    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-6rkmq"]
Nov 03 21:06:23.762708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.762676    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 21:06:23.853148 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853109    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-kube-api-access-rgb4x\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-kube-api-access-rgb4x\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.853351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853175    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-data\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.853351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853232    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-certs\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.853465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853350    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-credential-socket\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.853465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853393    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-envoy\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.853588 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853497    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70311607-242b-4964-ab22-0a73105e0ac0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70311607-242b-4964-ab22-0a73105e0ac0-istio-podinfo\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.853588 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853499    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-data" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:23.853588 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853533    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-certs" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:23.853588 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853571    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70311607-242b-4964-ab22-0a73105e0ac0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70311607-242b-4964-ab22-0a73105e0ac0-istiod-ca-cert\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.853810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853628    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-credential-socket" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:23.853810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853658    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-istio-token\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-istio-token\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.853810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853701    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-socket\") pod \"70311607-242b-4964-ab22-0a73105e0ac0\" (UID: \"70311607-242b-4964-ab22-0a73105e0ac0\") "
Nov 03 21:06:23.854029 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.853958    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70311607-242b-4964-ab22-0a73105e0ac0-istiod-ca-cert" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:23.854341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.854264    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-socket" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:23.854670 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.854649    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.854712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.854678    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.854712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.854694    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.854776 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.854708    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.854776 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.854722    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70311607-242b-4964-ab22-0a73105e0ac0-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.855851 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.855827    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-envoy" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:23.855967 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.855941    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-istio-token" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:23.856176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.856153    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70311607-242b-4964-ab22-0a73105e0ac0-istio-podinfo" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:06:23.856276 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.856213    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-kube-api-access-rgb4x" pod "70311607-242b-4964-ab22-0a73105e0ac0" (UID: "70311607-242b-4964-ab22-0a73105e0ac0"). InnerVolumeSpecName "kube-api-access-rgb4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:23.955980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.955902    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.955980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.955930    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgb4x\" (UniqueName: \"kubernetes.io/projected/70311607-242b-4964-ab22-0a73105e0ac0-kube-api-access-rgb4x\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.955980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.955940    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70311607-242b-4964-ab22-0a73105e0ac0-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.955980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:23.955951    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70311607-242b-4964-ab22-0a73105e0ac0-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:24.111086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.111048    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103c0f10-3b8e-4747-b25a-84634edc6a3a" path="/var/lib/kubelet/pods/103c0f10-3b8e-4747-b25a-84634edc6a3a/volumes"
Nov 03 21:06:24.111648 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.111631    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" path="/var/lib/kubelet/pods/7f17ae4d-acf3-4475-b7d2-1c9e8704a0de/volumes"
Nov 03 21:06:24.450708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.450667    5196 generic.go:356] "Generic (PLEG): container finished" podID="70311607-242b-4964-ab22-0a73105e0ac0" containerID="30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20" exitCode=0
Nov 03 21:06:24.450906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.450760    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" event={"ID":"70311607-242b-4964-ab22-0a73105e0ac0","Type":"ContainerDied","Data":"30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20"}
Nov 03 21:06:24.450906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.450783    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"
Nov 03 21:06:24.451036 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.450906    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96" event={"ID":"70311607-242b-4964-ab22-0a73105e0ac0","Type":"ContainerDied","Data":"a06b4f0378c986610571f013e3af4031938d22574a79fa5598674d7db8163114"}
Nov 03 21:06:24.451036 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.450949    5196 scope.go:122] "RemoveContainer" containerID="30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20"
Nov 03 21:06:24.465589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.465568    5196 scope.go:122] "RemoveContainer" containerID="30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20"
Nov 03 21:06:24.465892 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:06:24.465872    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20\": container with ID starting with 30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20 not found: ID does not exist" containerID="30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20"
Nov 03 21:06:24.465942 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.465901    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20"} err="failed to get container status \"30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20\": rpc error: code = NotFound desc = could not find container \"30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20\": container with ID starting with 30eadef29cb2c423631158b995d3ad0e112efb567b8cd9f6717a2737b8d62c20 not found: ID does not exist"
Nov 03 21:06:24.608290 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.608254    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"]
Nov 03 21:06:24.655993 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:24.655958    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-ctl96"]
Nov 03 21:06:25.873686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:25.873651    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:26.110899 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:26.110864    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70311607-242b-4964-ab22-0a73105e0ac0" path="/var/lib/kubelet/pods/70311607-242b-4964-ab22-0a73105e0ac0/volumes"
Nov 03 21:06:26.139118 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:26.139024    5196 scope.go:122] "RemoveContainer" containerID="85163c644e145cf696074fcf96b6b654b0f236001799225c35473e389e52a7f7"
Nov 03 21:06:30.546065 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:30.546023    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:33.486120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:33.486068    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:06:35.302912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:35.302875    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:35.813550 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:35.813494    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:38.969528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:38.969431    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:39.122667 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:39.122616    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:39.602118 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:39.602018    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:41.298605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:41.298565    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:41.489896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:41.489856    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:41.654291 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:41.654246    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:41.692743 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:41.692705    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"]
Nov 03 21:06:41.740666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:41.740632    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c63dc3-5jpch"]
Nov 03 21:06:42.112941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:42.112895    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21351d9d-6d91-4e0e-8a2f-b39db70ed597" path="/var/lib/kubelet/pods/21351d9d-6d91-4e0e-8a2f-b39db70ed597/volumes"
Nov 03 21:06:42.147384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:42.147350    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:46.659301 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:46.659268    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:47.922858 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:47.922821    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:48.118249 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:48.118217    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:48.280184 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:48.280064    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:48.436712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:48.436669    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:48.491690 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:06:48.491654    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:06:49.304313 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:49.304265    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:49.508251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:49.508214    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:51.508553 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:51.508517    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:51.710334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:51.710297    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:56.293875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:56.293791    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:56.489036 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:56.488999    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:56.633583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:06:56.633549    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:00.233187 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:00.233131    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:00.533816 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:00.533705    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:01.163699 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:01.163655    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:05.819699 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:05.819666    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:06.010607 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:06.010565    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:06.158972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:06.158881    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:10.760805 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:10.760764    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:15.591645 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:15.591606    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:18.486892 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:07:18.486846    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:07:20.280041 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:20.279992    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:21.947083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:21.947043    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:24.563260 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.563216    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"]
Nov 03 21:07:24.564009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.563993    5196 cpu_manager.go:414] "containerMap: removing container" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy"
Nov 03 21:07:24.564009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564009    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy"
Nov 03 21:07:24.564133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564051    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy"
Nov 03 21:07:24.564133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564056    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy"
Nov 03 21:07:24.564133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564074    5196 cpu_manager.go:414] "containerMap: removing container" podUID="298784fb-3289-4967-86cf-1b69c65d6a68" containerName="revoke-keys"
Nov 03 21:07:24.564133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564080    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298784fb-3289-4967-86cf-1b69c65d6a68" containerName="revoke-keys"
Nov 03 21:07:24.564133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564105    5196 cpu_manager.go:414] "containerMap: removing container" podUID="103c0f10-3b8e-4747-b25a-84634edc6a3a" containerName="discovery"
Nov 03 21:07:24.564133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564110    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103c0f10-3b8e-4747-b25a-84634edc6a3a" containerName="discovery"
Nov 03 21:07:24.564325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564218    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="298784fb-3289-4967-86cf-1b69c65d6a68" containerName="revoke-keys"
Nov 03 21:07:24.564325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564225    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="103c0f10-3b8e-4747-b25a-84634edc6a3a" containerName="discovery"
Nov 03 21:07:24.564325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564235    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="70311607-242b-4964-ab22-0a73105e0ac0" containerName="istio-proxy"
Nov 03 21:07:24.564325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.564246    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f17ae4d-acf3-4475-b7d2-1c9e8704a0de" containerName="istio-proxy"
Nov 03 21:07:24.568365 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.568339    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.619057 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.619020    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"]
Nov 03 21:07:24.660960 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.660927    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:07:24.661168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.660992    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:07:24.664736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.664707    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmr4\" (UniqueName: \"kubernetes.io/projected/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-kube-api-access-pnmr4\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.664915 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.664896    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.665142 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.665119    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.766454 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.766413    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.766677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.766583    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmr4\" (UniqueName: \"kubernetes.io/projected/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-kube-api-access-pnmr4\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.766751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.766739    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.767368 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.767346    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.769048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.769023    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.821012 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.820970    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmr4\" (UniqueName: \"kubernetes.io/projected/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-kube-api-access-pnmr4\") pod \"maas-api-revoke-sub-869da0d9-lksww\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:24.883778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:24.883748    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:25.029849 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:25.029810    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:25.267924 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:25.267831    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"]
Nov 03 21:07:25.274066 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:07:25.274026    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cc87a0_5cf0_4b7c_a72b_75fa1435b92b.slice/crio-41ff1146bb2f8fbf900c613610bcdd5ee1f2315eedfc2123ddaca1530ca7cb64 WatchSource:0}: Error finding container 41ff1146bb2f8fbf900c613610bcdd5ee1f2315eedfc2123ddaca1530ca7cb64: Status 404 returned error can't find the container with id 41ff1146bb2f8fbf900c613610bcdd5ee1f2315eedfc2123ddaca1530ca7cb64
Nov 03 21:07:26.081348 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:26.081313    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww" event={"ID":"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b","Type":"ContainerStarted","Data":"e4761a453badeba61606cfe414c760138c00324f9f5040b40dea7bb1345a3630"}
Nov 03 21:07:26.081768 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:26.081407    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww" event={"ID":"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b","Type":"ContainerStarted","Data":"41ff1146bb2f8fbf900c613610bcdd5ee1f2315eedfc2123ddaca1530ca7cb64"}
Nov 03 21:07:26.190167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:26.190114    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww" podStartSLOduration=2.190081441 podStartE2EDuration="2.190081441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:26.187901293 +0000 UTC m=+2286.191860024" watchObservedRunningTime="2026-11-03 21:07:26.190081441 +0000 UTC m=+2286.194040197"
Nov 03 21:07:27.091481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:27.091446    5196 generic.go:356] "Generic (PLEG): container finished" podID="51cc87a0-5cf0-4b7c-a72b-75fa1435b92b" containerID="e4761a453badeba61606cfe414c760138c00324f9f5040b40dea7bb1345a3630" exitCode=0
Nov 03 21:07:27.091900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:27.091531    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww" event={"ID":"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b","Type":"ContainerDied","Data":"e4761a453badeba61606cfe414c760138c00324f9f5040b40dea7bb1345a3630"}
Nov 03 21:07:27.679251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:27.679213    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:28.277117 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.277069    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:28.328580 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.328546    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-openshift-ca\") pod \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") "
Nov 03 21:07:28.328771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.328685    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-xks-ca\") pod \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") "
Nov 03 21:07:28.328863 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.328845    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-kube-api-access-pnmr4\" (UniqueName: \"kubernetes.io/projected/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-kube-api-access-pnmr4\") pod \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\" (UID: \"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b\") "
Nov 03 21:07:28.328973 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.328953    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-openshift-ca" pod "51cc87a0-5cf0-4b7c-a72b-75fa1435b92b" (UID: "51cc87a0-5cf0-4b7c-a72b-75fa1435b92b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:07:28.329461 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.329402    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:28.330946 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.330921    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-xks-ca" pod "51cc87a0-5cf0-4b7c-a72b-75fa1435b92b" (UID: "51cc87a0-5cf0-4b7c-a72b-75fa1435b92b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:28.331062 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.331042    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-kube-api-access-pnmr4" pod "51cc87a0-5cf0-4b7c-a72b-75fa1435b92b" (UID: "51cc87a0-5cf0-4b7c-a72b-75fa1435b92b"). InnerVolumeSpecName "kube-api-access-pnmr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:28.431284 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.431185    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:28.431284 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:28.431219    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnmr4\" (UniqueName: \"kubernetes.io/projected/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b-kube-api-access-pnmr4\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:29.111013 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:29.110980    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"
Nov 03 21:07:29.111213 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:29.111016    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww" event={"ID":"51cc87a0-5cf0-4b7c-a72b-75fa1435b92b","Type":"ContainerDied","Data":"41ff1146bb2f8fbf900c613610bcdd5ee1f2315eedfc2123ddaca1530ca7cb64"}
Nov 03 21:07:29.111213 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:29.111075    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41ff1146bb2f8fbf900c613610bcdd5ee1f2315eedfc2123ddaca1530ca7cb64"
Nov 03 21:07:29.775826 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:29.775795    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:34.464836 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:34.464794    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:35.418155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:35.418117    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:39.169949 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:39.169909    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:41.325519 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:41.325479    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:42.187084 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:42.187039    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:07:43.903110 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:43.903058    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:48.487213 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:07:48.487172    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:07:48.613226 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:48.613185    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:48.760970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:48.760867    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:53.371692 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:53.371643    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:58.161737 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:07:58.161698    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:02.619393 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.619339    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"]
Nov 03 21:08:02.620164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.620147    5196 cpu_manager.go:414] "containerMap: removing container" podUID="51cc87a0-5cf0-4b7c-a72b-75fa1435b92b" containerName="revoke-keys"
Nov 03 21:08:02.620228 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.620164    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cc87a0-5cf0-4b7c-a72b-75fa1435b92b" containerName="revoke-keys"
Nov 03 21:08:02.620306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.620297    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cc87a0-5cf0-4b7c-a72b-75fa1435b92b" containerName="revoke-keys"
Nov 03 21:08:02.624378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.624357    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.673528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.673490    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-211d68bf-openshift-default-dockercfg-c9mrc\""
Nov 03 21:08:02.772014 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.771970    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wjc\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-kube-api-access-m4wjc\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.772237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.772135    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.772237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.772202    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.772237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.772227    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.772447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.772286    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.772447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.772337    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.772447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.772420    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.772627 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.772556    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.772627 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.772606    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.812453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.812416    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"]
Nov 03 21:08:02.874642 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.874558    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.874642 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.874621    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.874880 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.874649    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.874880 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.874685    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.874880 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.874733    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.874999 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.874889    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.875169 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.875147    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.875275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.875162    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.875275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.875194    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.875275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.875216    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.875465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.875308    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wjc\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-kube-api-access-m4wjc\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.875465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.875440    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.875558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.875472    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.875669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.875645    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.877402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.877380    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.877748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.877730    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.929319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.929282    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.929501 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.929467    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wjc\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-kube-api-access-m4wjc\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-v5lgq\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.939378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.939346    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:02.957604 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:02.957575    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:03.408420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:03.408388    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"]
Nov 03 21:08:03.413928 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:08:03.413890    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2613d8e3_3f14_4278_9b00_30dbfd9246d8.slice/crio-668fcf5cdbab30a7175d64e17732e368987d47662e2ed6551fcd919c4b5cd778 WatchSource:0}: Error finding container 668fcf5cdbab30a7175d64e17732e368987d47662e2ed6551fcd919c4b5cd778: Status 404 returned error can't find the container with id 668fcf5cdbab30a7175d64e17732e368987d47662e2ed6551fcd919c4b5cd778
Nov 03 21:08:03.416710 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:03.416674    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:03.416802 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:03.416750    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:03.416802 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:03.416780    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:04.426375 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:04.426307    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" event={"ID":"2613d8e3-3f14-4278-9b00-30dbfd9246d8","Type":"ContainerStarted","Data":"1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61"}
Nov 03 21:08:04.426873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:04.426397    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" event={"ID":"2613d8e3-3f14-4278-9b00-30dbfd9246d8","Type":"ContainerStarted","Data":"668fcf5cdbab30a7175d64e17732e368987d47662e2ed6551fcd919c4b5cd778"}
Nov 03 21:08:04.588375 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:04.588323    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" podStartSLOduration=2.588305307 podStartE2EDuration="2.588305307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:04.583511178 +0000 UTC m=+2324.587469937" watchObservedRunningTime="2026-11-03 21:08:04.588305307 +0000 UTC m=+2324.592264031"
Nov 03 21:08:04.952552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:04.952513    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:04.960772 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:04.960720    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 21:08:04.961054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:04.960808    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 21:08:05.957645 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:05.957602    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 21:08:05.958341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:05.958307    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 21:08:06.953119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:06.941532    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 21:08:06.953119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:06.941602    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 21:08:07.830215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:07.830177    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:07.951122 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:07.947768    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 21:08:07.951122 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:07.947833    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 21:08:08.961289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:08.961232    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 21:08:08.961800 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:08.961329    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 21:08:09.952120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:09.943801    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 21:08:09.952120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:09.943960    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 21:08:10.973920 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:10.973879    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:10.976080 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:10.976054    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:10.985413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:10.985387    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:08:12.008751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:12.008704    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:12.465060 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:12.465020    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:13.162733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:13.162656    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:13.329806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:13.329751    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:13.515564 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:13.515475    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:17.735045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.734999    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"]
Nov 03 21:08:17.745437 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.745400    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.829414 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.829378    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.829414 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.829421    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18a218-f0e9-4880-b7b9-cce51ac03077-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.829647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.829588    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.829719 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.829651    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.829820 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.829797    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.829931 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.829916    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.829992 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.829969    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsxt\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-kube-api-access-tnsxt\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.830045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.830023    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.830188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.830167    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931409    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931507    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931543    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsxt\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-kube-api-access-tnsxt\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931625    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931791 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931698    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931801    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931830    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18a218-f0e9-4880-b7b9-cce51ac03077-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931948 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931892    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931948 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931894    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.931948 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.931924    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.932163 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.932060    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.932327 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.932297    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.932409 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.932326    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.932641 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.932620    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18a218-f0e9-4880-b7b9-cce51ac03077-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.934368 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.934338    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.934562 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.934543    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.943375 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.943342    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"]
Nov 03 21:08:17.986496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.986413    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:17.986670 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:17.986524    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsxt\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-kube-api-access-tnsxt\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-lncvp\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:18.063985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.063942    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:18.092161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.092117    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:18.268777 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.268681    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:18.441122 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.433757    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:18.516681 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:08:18.516638    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:08:18.522390 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.522304    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"]
Nov 03 21:08:18.545569 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:08:18.537592    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad18a218_f0e9_4880_b7b9_cce51ac03077.slice/crio-cad469ad9514a0251fe25e746e7387b55be745339b8caba7967df21b9059c84b WatchSource:0}: Error finding container cad469ad9514a0251fe25e746e7387b55be745339b8caba7967df21b9059c84b: Status 404 returned error can't find the container with id cad469ad9514a0251fe25e746e7387b55be745339b8caba7967df21b9059c84b
Nov 03 21:08:18.545784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.543249    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:18.545979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.545953    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:18.550432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.546624    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:18.626961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:18.626921    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" event={"ID":"ad18a218-f0e9-4880-b7b9-cce51ac03077","Type":"ContainerStarted","Data":"cad469ad9514a0251fe25e746e7387b55be745339b8caba7967df21b9059c84b"}
Nov 03 21:08:19.663083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:19.663038    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" event={"ID":"ad18a218-f0e9-4880-b7b9-cce51ac03077","Type":"ContainerStarted","Data":"816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9"}
Nov 03 21:08:19.792040 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:19.791990    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" podStartSLOduration=2.791975379 podStartE2EDuration="2.791975379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:19.788597804 +0000 UTC m=+2339.792556561" watchObservedRunningTime="2026-11-03 21:08:19.791975379 +0000 UTC m=+2339.795934103"
Nov 03 21:08:19.866784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:19.866742    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7280-3a3f8c"]
Nov 03 21:08:19.871877 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:19.871849    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:19.901214 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:19.901180    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7280-3a3f8c"]
Nov 03 21:08:19.921566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:19.921476    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-24vlg\""
Nov 03 21:08:19.974538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:19.974499    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r2rd\" (UniqueName: \"kubernetes.io/projected/5b432008-5a8a-4805-b9f2-6dbf81ecc317-kube-api-access-2r2rd\") pod \"test-curl-7280-3a3f8c\" (UID: \"5b432008-5a8a-4805-b9f2-6dbf81ecc317\") " pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:20.064491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:20.064446    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:20.066648 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:20.066619    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lncvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 03 21:08:20.066801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:20.066667    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 03 21:08:20.075689 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:20.075654    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r2rd\" (UniqueName: \"kubernetes.io/projected/5b432008-5a8a-4805-b9f2-6dbf81ecc317-kube-api-access-2r2rd\") pod \"test-curl-7280-3a3f8c\" (UID: \"5b432008-5a8a-4805-b9f2-6dbf81ecc317\") " pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:20.129360 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:20.129327    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r2rd\" (UniqueName: \"kubernetes.io/projected/5b432008-5a8a-4805-b9f2-6dbf81ecc317-kube-api-access-2r2rd\") pod \"test-curl-7280-3a3f8c\" (UID: \"5b432008-5a8a-4805-b9f2-6dbf81ecc317\") " pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:20.257179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:20.256065    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-24vlg\""
Nov 03 21:08:20.259388 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:20.259226    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:20.664198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:20.664157    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7280-3a3f8c"]
Nov 03 21:08:20.664828 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:08:20.664797    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b432008_5a8a_4805_b9f2_6dbf81ecc317.slice/crio-d7d29648f154e0ee20120d4a4b3a2459a869db8fca05a274948595092ab975c2 WatchSource:0}: Error finding container d7d29648f154e0ee20120d4a4b3a2459a869db8fca05a274948595092ab975c2: Status 404 returned error can't find the container with id d7d29648f154e0ee20120d4a4b3a2459a869db8fca05a274948595092ab975c2
Nov 03 21:08:21.065271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.065170    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lncvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 03 21:08:21.065271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.065244    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 03 21:08:21.364757 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.364721    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:21.595484 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.595433    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:21.687203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.687110    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7280-3a3f8c" event={"ID":"5b432008-5a8a-4805-b9f2-6dbf81ecc317","Type":"ContainerStarted","Data":"42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01"}
Nov 03 21:08:21.687203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.687159    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7280-3a3f8c" event={"ID":"5b432008-5a8a-4805-b9f2-6dbf81ecc317","Type":"ContainerStarted","Data":"d7d29648f154e0ee20120d4a4b3a2459a869db8fca05a274948595092ab975c2"}
Nov 03 21:08:21.791216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.791168    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"]
Nov 03 21:08:21.802664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.802631    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:21.809595 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.809332    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7280-3a3f8c" podStartSLOduration=2.809313947 podStartE2EDuration="2.809313947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:21.807480573 +0000 UTC m=+2341.811439298" watchObservedRunningTime="2026-11-03 21:08:21.809313947 +0000 UTC m=+2341.813272681"
Nov 03 21:08:21.824764 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.824734    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"]
Nov 03 21:08:21.883451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.881972    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-562eb2-dockercfg-4gjm5\""
Nov 03 21:08:21.883451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.882130    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-562eb2\""
Nov 03 21:08:21.932157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.927030    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"]
Nov 03 21:08:21.946176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:21.933709    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:22.067120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.065134    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681aac0-0c6c-4dec-b8ea-9b837683c320-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c\" (UID: \"f681aac0-0c6c-4dec-b8ea-9b837683c320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:22.067120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.065233    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fc4bb-d514-46e7-98a0-f457940b468f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx\" (UID: \"898fc4bb-d514-46e7-98a0-f457940b468f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:22.067120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.065376    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxq9\" (UniqueName: \"kubernetes.io/projected/f681aac0-0c6c-4dec-b8ea-9b837683c320-kube-api-access-wnxq9\") pod \"payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c\" (UID: \"f681aac0-0c6c-4dec-b8ea-9b837683c320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:22.067120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.065455    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4jq\" (UniqueName: \"kubernetes.io/projected/898fc4bb-d514-46e7-98a0-f457940b468f-kube-api-access-5b4jq\") pod \"payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx\" (UID: \"898fc4bb-d514-46e7-98a0-f457940b468f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:22.108566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.107491    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lncvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 03 21:08:22.108566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.107565    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 03 21:08:22.126213 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.126127    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"]
Nov 03 21:08:22.169411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.168138    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxq9\" (UniqueName: \"kubernetes.io/projected/f681aac0-0c6c-4dec-b8ea-9b837683c320-kube-api-access-wnxq9\") pod \"payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c\" (UID: \"f681aac0-0c6c-4dec-b8ea-9b837683c320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:22.169411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.168275    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4jq\" (UniqueName: \"kubernetes.io/projected/898fc4bb-d514-46e7-98a0-f457940b468f-kube-api-access-5b4jq\") pod \"payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx\" (UID: \"898fc4bb-d514-46e7-98a0-f457940b468f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:22.169411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.168404    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681aac0-0c6c-4dec-b8ea-9b837683c320-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c\" (UID: \"f681aac0-0c6c-4dec-b8ea-9b837683c320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:22.169411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.168497    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fc4bb-d514-46e7-98a0-f457940b468f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx\" (UID: \"898fc4bb-d514-46e7-98a0-f457940b468f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:22.169869 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.169619    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fc4bb-d514-46e7-98a0-f457940b468f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx\" (UID: \"898fc4bb-d514-46e7-98a0-f457940b468f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:22.170037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.170009    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681aac0-0c6c-4dec-b8ea-9b837683c320-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c\" (UID: \"f681aac0-0c6c-4dec-b8ea-9b837683c320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:22.226885 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.226812    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxq9\" (UniqueName: \"kubernetes.io/projected/f681aac0-0c6c-4dec-b8ea-9b837683c320-kube-api-access-wnxq9\") pod \"payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c\" (UID: \"f681aac0-0c6c-4dec-b8ea-9b837683c320\") " pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:22.227110 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.226950    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4jq\" (UniqueName: \"kubernetes.io/projected/898fc4bb-d514-46e7-98a0-f457940b468f-kube-api-access-5b4jq\") pod \"payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx\" (UID: \"898fc4bb-d514-46e7-98a0-f457940b468f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:22.426157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.426105    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:22.428248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.428213    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:22.804480 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:22.804185    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:23.074499 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.074285    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lncvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 03 21:08:23.074499 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.074345    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 03 21:08:23.128591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.128487    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"]
Nov 03 21:08:23.161170 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.155661    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"]
Nov 03 21:08:23.510621 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.510585    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7280-3a3f8c"]
Nov 03 21:08:23.518149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.517480    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7280-3a3f8c"]
Nov 03 21:08:23.867757 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.867702    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c" event={"ID":"f681aac0-0c6c-4dec-b8ea-9b837683c320","Type":"ContainerStarted","Data":"cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10"}
Nov 03 21:08:23.868324 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.867778    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c" event={"ID":"f681aac0-0c6c-4dec-b8ea-9b837683c320","Type":"ContainerStarted","Data":"663776cd6fcfc8faadd8c2997dfbfd3e8351d5373805ce3a24e54096a0b08ab5"}
Nov 03 21:08:23.868324 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.867810    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:23.870301 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.870270    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx" event={"ID":"898fc4bb-d514-46e7-98a0-f457940b468f","Type":"ContainerStarted","Data":"70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b"}
Nov 03 21:08:23.870479 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.870331    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx" event={"ID":"898fc4bb-d514-46e7-98a0-f457940b468f","Type":"ContainerStarted","Data":"b8cfdef09f1daad00c17618f4d68dc16e6e6feb88d40825085a12f6179698b10"}
Nov 03 21:08:23.870479 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.870413    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:23.870565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.870533    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7280-3a3f8c" podUID="5b432008-5a8a-4805-b9f2-6dbf81ecc317" containerName="test-curl-7280-3a3f8c" containerID="cri-o://42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01" gracePeriod=2
Nov 03 21:08:23.976742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:23.976637    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c" podStartSLOduration=2.9766231850000002 podStartE2EDuration="2.976623185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:23.974623041 +0000 UTC m=+2343.978581788" watchObservedRunningTime="2026-11-03 21:08:23.976623185 +0000 UTC m=+2343.980581908"
Nov 03 21:08:24.064837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.064799    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-lncvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 03 21:08:24.065025 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.064854    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 03 21:08:24.085136 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.085046    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx" podStartSLOduration=3.085024385 podStartE2EDuration="3.085024385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:24.081186315 +0000 UTC m=+2344.085145061" watchObservedRunningTime="2026-11-03 21:08:24.085024385 +0000 UTC m=+2344.088983109"
Nov 03 21:08:24.099326 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.098237    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"]
Nov 03 21:08:24.099559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.099519    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5b432008-5a8a-4805-b9f2-6dbf81ecc317" containerName="test-curl-7280-3a3f8c"
Nov 03 21:08:24.099559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.099540    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b432008-5a8a-4805-b9f2-6dbf81ecc317" containerName="test-curl-7280-3a3f8c"
Nov 03 21:08:24.099808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.099779    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b432008-5a8a-4805-b9f2-6dbf81ecc317" containerName="test-curl-7280-3a3f8c"
Nov 03 21:08:24.108937 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.104573    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.130214 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:08:24.129328    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-3a3f8c\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="5b432008-5a8a-4805-b9f2-6dbf81ecc317" pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:24.164140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.163382    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:24.164140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.163560    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:08:24.173570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.172913    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.173570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.173201    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4xfw\" (UniqueName: \"kubernetes.io/projected/b01ddc62-1b47-4f82-a746-78f3eec64c6d-kube-api-access-k4xfw\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.173570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.173408    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.184119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.184064    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"]
Nov 03 21:08:24.277571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.277523    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.279754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.279709    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.279754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.279038    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.280059 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.280022    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4xfw\" (UniqueName: \"kubernetes.io/projected/b01ddc62-1b47-4f82-a746-78f3eec64c6d-kube-api-access-k4xfw\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.288162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.283974    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.351178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.334486    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4xfw\" (UniqueName: \"kubernetes.io/projected/b01ddc62-1b47-4f82-a746-78f3eec64c6d-kube-api-access-k4xfw\") pod \"maas-api-revoke-keys-e2e-mt-211d68bf-59cvl\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.431124 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.430996    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:24.647191 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.643861    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:24.807300 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.807203    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:24.847783 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.845928    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"]
Nov 03 21:08:24.853584 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:08:24.853553    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01ddc62_1b47_4f82_a746_78f3eec64c6d.slice/crio-d89ca6db85d18b48dc0dabaf7518eb06ade4fa4a694fbedbde82b48008f296f0 WatchSource:0}: Error finding container d89ca6db85d18b48dc0dabaf7518eb06ade4fa4a694fbedbde82b48008f296f0: Status 404 returned error can't find the container with id d89ca6db85d18b48dc0dabaf7518eb06ade4fa4a694fbedbde82b48008f296f0
Nov 03 21:08:24.885825 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:24.885796    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl" event={"ID":"b01ddc62-1b47-4f82-a746-78f3eec64c6d","Type":"ContainerStarted","Data":"d89ca6db85d18b48dc0dabaf7518eb06ade4fa4a694fbedbde82b48008f296f0"}
Nov 03 21:08:25.082244 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:25.082203    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:25.262686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:25.262647    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"]
Nov 03 21:08:25.279425 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:25.279398    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:25.279905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:25.279872    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:25.280794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:25.280775    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:08:25.309296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:25.309262    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-46ca0b-8ds7m"]
Nov 03 21:08:25.708586 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:25.708498    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:25.904828 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:25.904787    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl" event={"ID":"b01ddc62-1b47-4f82-a746-78f3eec64c6d","Type":"ContainerStarted","Data":"5c7f4d70bb246b7816229acc245065d5b25c5aa30f7051f64e266b2142fd8140"}
Nov 03 21:08:26.014342 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.014230    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl" podStartSLOduration=2.014215273 podStartE2EDuration="2.014215273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:26.01255753 +0000 UTC m=+2346.016516254" watchObservedRunningTime="2026-11-03 21:08:26.014215273 +0000 UTC m=+2346.018173996"
Nov 03 21:08:26.112274 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.112240    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f3bdbb-a243-47f8-ad08-de63e4ccf5ec" path="/var/lib/kubelet/pods/22f3bdbb-a243-47f8-ad08-de63e4ccf5ec/volumes"
Nov 03 21:08:26.127452 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.127415    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:26.142294 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.142265    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:26.217360 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.217328    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b432008-5a8a-4805-b9f2-6dbf81ecc317-kube-api-access-2r2rd\" (UniqueName: \"kubernetes.io/projected/5b432008-5a8a-4805-b9f2-6dbf81ecc317-kube-api-access-2r2rd\") pod \"5b432008-5a8a-4805-b9f2-6dbf81ecc317\" (UID: \"5b432008-5a8a-4805-b9f2-6dbf81ecc317\") "
Nov 03 21:08:26.220563 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.220522    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b432008-5a8a-4805-b9f2-6dbf81ecc317-kube-api-access-2r2rd" pod "5b432008-5a8a-4805-b9f2-6dbf81ecc317" (UID: "5b432008-5a8a-4805-b9f2-6dbf81ecc317"). InnerVolumeSpecName "kube-api-access-2r2rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:26.319768 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.319726    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r2rd\" (UniqueName: \"kubernetes.io/projected/5b432008-5a8a-4805-b9f2-6dbf81ecc317-kube-api-access-2r2rd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:26.550802 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.550717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:26.921770 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.921725    5196 generic.go:356] "Generic (PLEG): container finished" podID="b01ddc62-1b47-4f82-a746-78f3eec64c6d" containerID="5c7f4d70bb246b7816229acc245065d5b25c5aa30f7051f64e266b2142fd8140" exitCode=0
Nov 03 21:08:26.922304 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.921883    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl" event={"ID":"b01ddc62-1b47-4f82-a746-78f3eec64c6d","Type":"ContainerDied","Data":"5c7f4d70bb246b7816229acc245065d5b25c5aa30f7051f64e266b2142fd8140"}
Nov 03 21:08:26.925209 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.925172    5196 generic.go:356] "Generic (PLEG): container finished" podID="5b432008-5a8a-4805-b9f2-6dbf81ecc317" containerID="42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01" exitCode=137
Nov 03 21:08:26.925426 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.925222    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:26.925572 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.925546    5196 scope.go:122] "RemoveContainer" containerID="42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01"
Nov 03 21:08:26.942902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.942876    5196 scope.go:122] "RemoveContainer" containerID="42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01"
Nov 03 21:08:26.943281 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:08:26.943260    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01\": container with ID starting with 42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01 not found: ID does not exist" containerID="42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01"
Nov 03 21:08:26.943344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:26.943292    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01"} err="failed to get container status \"42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01\": rpc error: code = NotFound desc = could not find container \"42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01\": container with ID starting with 42d324f821c640d0703ca75c3f56a952d0fe4e65fab3b68f32e983ce517c3b01 not found: ID does not exist"
Nov 03 21:08:27.092131 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:08:27.088360    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-3a3f8c\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="5b432008-5a8a-4805-b9f2-6dbf81ecc317" pod="opendatahub/test-curl-7280-3a3f8c"
Nov 03 21:08:27.357842 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:27.357797    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:27.813827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:27.813721    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:28.113738 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.113709    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b432008-5a8a-4805-b9f2-6dbf81ecc317" path="/var/lib/kubelet/pods/5b432008-5a8a-4805-b9f2-6dbf81ecc317/volumes"
Nov 03 21:08:28.120707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.120681    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:28.152292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.152255    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-xks-ca\") pod \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") "
Nov 03 21:08:28.152601 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.152568    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01ddc62-1b47-4f82-a746-78f3eec64c6d-kube-api-access-k4xfw\" (UniqueName: \"kubernetes.io/projected/b01ddc62-1b47-4f82-a746-78f3eec64c6d-kube-api-access-k4xfw\") pod \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") "
Nov 03 21:08:28.152675 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.152659    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-openshift-ca\") pod \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\" (UID: \"b01ddc62-1b47-4f82-a746-78f3eec64c6d\") "
Nov 03 21:08:28.153027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.153001    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-openshift-ca" pod "b01ddc62-1b47-4f82-a746-78f3eec64c6d" (UID: "b01ddc62-1b47-4f82-a746-78f3eec64c6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:28.153421 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.153399    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:28.154823 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.154803    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-xks-ca" pod "b01ddc62-1b47-4f82-a746-78f3eec64c6d" (UID: "b01ddc62-1b47-4f82-a746-78f3eec64c6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:28.154893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.154804    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01ddc62-1b47-4f82-a746-78f3eec64c6d-kube-api-access-k4xfw" pod "b01ddc62-1b47-4f82-a746-78f3eec64c6d" (UID: "b01ddc62-1b47-4f82-a746-78f3eec64c6d"). InnerVolumeSpecName "kube-api-access-k4xfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:28.254861 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.254827    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4xfw\" (UniqueName: \"kubernetes.io/projected/b01ddc62-1b47-4f82-a746-78f3eec64c6d-kube-api-access-k4xfw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:28.254861 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.254858    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01ddc62-1b47-4f82-a746-78f3eec64c6d-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:28.947526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.947494    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"
Nov 03 21:08:28.947526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.947516    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl" event={"ID":"b01ddc62-1b47-4f82-a746-78f3eec64c6d","Type":"ContainerDied","Data":"d89ca6db85d18b48dc0dabaf7518eb06ade4fa4a694fbedbde82b48008f296f0"}
Nov 03 21:08:28.947819 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:28.947569    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d89ca6db85d18b48dc0dabaf7518eb06ade4fa4a694fbedbde82b48008f296f0"
Nov 03 21:08:29.661663 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:29.661629    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:31.551235 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:31.551194    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:31.739301 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:31.739266    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:31.906664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:31.906631    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:32.697683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.697648    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"]
Nov 03 21:08:32.698554 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.698535    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b01ddc62-1b47-4f82-a746-78f3eec64c6d" containerName="revoke-keys"
Nov 03 21:08:32.698612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.698556    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01ddc62-1b47-4f82-a746-78f3eec64c6d" containerName="revoke-keys"
Nov 03 21:08:32.698688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.698678    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01ddc62-1b47-4f82-a746-78f3eec64c6d" containerName="revoke-keys"
Nov 03 21:08:32.702735 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.702718    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.755070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.755036    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"]
Nov 03 21:08:32.835615 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.835576    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"]
Nov 03 21:08:32.840854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.840821    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.841780 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.841750    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.841889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.841872    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mzq\" (UniqueName: \"kubernetes.io/projected/afccfc6a-322c-433a-87c0-65e0bf0bf85e-kube-api-access-q8mzq\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.842184 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.842165    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.855957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.855931    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:32.856203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.855974    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:08:32.898113 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.898072    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"]
Nov 03 21:08:32.944052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944007    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.944052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944054    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.944293 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944191    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94587bc3-1388-4ef1-aab9-a16c547680df-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.944293 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944259    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94587bc3-1388-4ef1-aab9-a16c547680df-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.944409 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944290    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxfm\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-kube-api-access-vkxfm\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.944465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944431    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.944521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944486    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.944588 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944570    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.944653 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944632    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.944707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944683    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.944820 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944800    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mzq\" (UniqueName: \"kubernetes.io/projected/afccfc6a-322c-433a-87c0-65e0bf0bf85e-kube-api-access-q8mzq\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.944877 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.944837    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:32.945426 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.945398    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.947180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.947159    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:32.999325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:32.999251    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mzq\" (UniqueName: \"kubernetes.io/projected/afccfc6a-322c-433a-87c0-65e0bf0bf85e-kube-api-access-q8mzq\") pod \"maas-api-revoke-keys-e2e-api-b-c571ef-mmff6\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:33.017896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.017869    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:33.046168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046086    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046297 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046201    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046297 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046282    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046414 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046384    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046537    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046580    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046624    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94587bc3-1388-4ef1-aab9-a16c547680df-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046628    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-data\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046703    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94587bc3-1388-4ef1-aab9-a16c547680df-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.046898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.046746    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxfm\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-kube-api-access-vkxfm\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.048025 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.047044    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.048025 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.047249    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-certs\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.048025 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.047368    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-credential-socket\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.048025 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.047697    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94587bc3-1388-4ef1-aab9-a16c547680df-istiod-ca-cert\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.048773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.048749    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-envoy\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.050391 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.050350    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94587bc3-1388-4ef1-aab9-a16c547680df-istio-podinfo\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.107829 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.107553    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxfm\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-kube-api-access-vkxfm\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.111537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.111505    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-istio-token\") pod \"e2e-mt-211d68bf-openshift-default-588997c885-8zgjr\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") " pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.156651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.156615    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:33.406699 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.406656    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"]
Nov 03 21:08:33.411951 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:08:33.411921    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafccfc6a_322c_433a_87c0_65e0bf0bf85e.slice/crio-166ab5f5f7da65551d508328ba3bbd06759bcad54cf9a08850baf2a36fdab963 WatchSource:0}: Error finding container 166ab5f5f7da65551d508328ba3bbd06759bcad54cf9a08850baf2a36fdab963: Status 404 returned error can't find the container with id 166ab5f5f7da65551d508328ba3bbd06759bcad54cf9a08850baf2a36fdab963
Nov 03 21:08:33.548735 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.548694    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"]
Nov 03 21:08:33.637359 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:08:33.637317    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94587bc3_1388_4ef1_aab9_a16c547680df.slice/crio-dca0078a445118b85bef125c8e864a1554b873d89e7a4271ba474b833b0dc0c3 WatchSource:0}: Error finding container dca0078a445118b85bef125c8e864a1554b873d89e7a4271ba474b833b0dc0c3: Status 404 returned error can't find the container with id dca0078a445118b85bef125c8e864a1554b873d89e7a4271ba474b833b0dc0c3
Nov 03 21:08:33.640334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.640285    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:33.640420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.640359    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:33.640420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:33.640389    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:08:34.058028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.057989    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6" event={"ID":"afccfc6a-322c-433a-87c0-65e0bf0bf85e","Type":"ContainerStarted","Data":"e46ce919b76e9cf3c567f0752106406c13cd94c5180fd63a12c1055056660ffa"}
Nov 03 21:08:34.058453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.058053    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6" event={"ID":"afccfc6a-322c-433a-87c0-65e0bf0bf85e","Type":"ContainerStarted","Data":"166ab5f5f7da65551d508328ba3bbd06759bcad54cf9a08850baf2a36fdab963"}
Nov 03 21:08:34.060190 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.060163    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" event={"ID":"94587bc3-1388-4ef1-aab9-a16c547680df","Type":"ContainerStarted","Data":"7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e"}
Nov 03 21:08:34.060342 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.060218    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" event={"ID":"94587bc3-1388-4ef1-aab9-a16c547680df","Type":"ContainerStarted","Data":"dca0078a445118b85bef125c8e864a1554b873d89e7a4271ba474b833b0dc0c3"}
Nov 03 21:08:34.156776 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.156742    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:34.157710 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.157679    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:34.158604 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.158582    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:08:34.158661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.158630    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 03 21:08:34.171331 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.171280    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6" podStartSLOduration=2.17126567 podStartE2EDuration="2.17126567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:34.168236691 +0000 UTC m=+2354.172195430" watchObservedRunningTime="2026-11-03 21:08:34.17126567 +0000 UTC m=+2354.175224394"
Nov 03 21:08:34.285639 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.285569    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podStartSLOduration=2.285544164 podStartE2EDuration="2.285544164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:34.277553633 +0000 UTC m=+2354.281512389" watchObservedRunningTime="2026-11-03 21:08:34.285544164 +0000 UTC m=+2354.289502889"
Nov 03 21:08:34.889256 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.889215    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:08:34.889450 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:34.889309    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:08:35.078519 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:35.078478    5196 generic.go:356] "Generic (PLEG): container finished" podID="afccfc6a-322c-433a-87c0-65e0bf0bf85e" containerID="e46ce919b76e9cf3c567f0752106406c13cd94c5180fd63a12c1055056660ffa" exitCode=0
Nov 03 21:08:35.079200 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:35.078562    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6" event={"ID":"afccfc6a-322c-433a-87c0-65e0bf0bf85e","Type":"ContainerDied","Data":"e46ce919b76e9cf3c567f0752106406c13cd94c5180fd63a12c1055056660ffa"}
Nov 03 21:08:35.157079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:35.157042    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:08:35.158170 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:35.158124    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 03 21:08:35.862143 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:35.861749    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:36.083276 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.083239    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:36.161818 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.161725    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:08:36.162086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.162055    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 03 21:08:36.272987 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.272956    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:36.373306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.373273    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"]
Nov 03 21:08:36.377732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.377707    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:36.407886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.407851    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-openshift-ca\") pod \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") "
Nov 03 21:08:36.408076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.407998    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afccfc6a-322c-433a-87c0-65e0bf0bf85e-kube-api-access-q8mzq\" (UniqueName: \"kubernetes.io/projected/afccfc6a-322c-433a-87c0-65e0bf0bf85e-kube-api-access-q8mzq\") pod \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") "
Nov 03 21:08:36.408076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.408025    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-xks-ca\") pod \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\" (UID: \"afccfc6a-322c-433a-87c0-65e0bf0bf85e\") "
Nov 03 21:08:36.408351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.408321    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-openshift-ca" pod "afccfc6a-322c-433a-87c0-65e0bf0bf85e" (UID: "afccfc6a-322c-433a-87c0-65e0bf0bf85e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:36.408668 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.408650    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:36.410475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.410448    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-xks-ca" pod "afccfc6a-322c-433a-87c0-65e0bf0bf85e" (UID: "afccfc6a-322c-433a-87c0-65e0bf0bf85e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:36.410600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.410453    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afccfc6a-322c-433a-87c0-65e0bf0bf85e-kube-api-access-q8mzq" pod "afccfc6a-322c-433a-87c0-65e0bf0bf85e" (UID: "afccfc6a-322c-433a-87c0-65e0bf0bf85e"). InnerVolumeSpecName "kube-api-access-q8mzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:36.415812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.415750    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88bc35-sqxq4"]
Nov 03 21:08:36.510072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.510028    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8mzq\" (UniqueName: \"kubernetes.io/projected/afccfc6a-322c-433a-87c0-65e0bf0bf85e-kube-api-access-q8mzq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:36.510072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.510075    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afccfc6a-322c-433a-87c0-65e0bf0bf85e-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:36.761996 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:36.761739    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:37.098860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:37.098832    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"
Nov 03 21:08:37.099418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:37.098868    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6" event={"ID":"afccfc6a-322c-433a-87c0-65e0bf0bf85e","Type":"ContainerDied","Data":"166ab5f5f7da65551d508328ba3bbd06759bcad54cf9a08850baf2a36fdab963"}
Nov 03 21:08:37.099418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:37.098946    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="166ab5f5f7da65551d508328ba3bbd06759bcad54cf9a08850baf2a36fdab963"
Nov 03 21:08:37.129996 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:37.129963    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:37.182196 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:37.182161    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:08:37.182501 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:37.182473    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 03 21:08:37.289806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:37.289767    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:38.112855 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:38.112822    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298784fb-3289-4967-86cf-1b69c65d6a68" path="/var/lib/kubelet/pods/298784fb-3289-4967-86cf-1b69c65d6a68/volumes"
Nov 03 21:08:38.160820 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:38.160775    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:08:38.161055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:38.160860    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 03 21:08:39.160577 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:39.159994    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:08:39.160577 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:39.160063    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 03 21:08:40.157721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:40.157675    5196 patch_prober.go:29] interesting pod/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:08:40.157948 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:40.157739    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 03 21:08:40.638413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:40.638372    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:40.882053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:40.878942    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:41.101952 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:41.101914    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:41.328754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:41.328410    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:41.330464 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:41.329504    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:41.330464 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:41.330232    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:08:41.587815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:41.587761    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:43.160900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:43.160860    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:43.323734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:43.323697    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:43.505927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:43.505806    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:44.279407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.279375    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"]
Nov 03 21:08:44.280225 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.280208    5196 cpu_manager.go:414] "containerMap: removing container" podUID="afccfc6a-322c-433a-87c0-65e0bf0bf85e" containerName="revoke-keys"
Nov 03 21:08:44.280275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.280226    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afccfc6a-322c-433a-87c0-65e0bf0bf85e" containerName="revoke-keys"
Nov 03 21:08:44.280418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.280408    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="afccfc6a-322c-433a-87c0-65e0bf0bf85e" containerName="revoke-keys"
Nov 03 21:08:44.284378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.284360    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.335647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.335608    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"]
Nov 03 21:08:44.349378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.349337    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8lhd\" (UniqueName: \"kubernetes.io/projected/82f6b7ac-cd4e-404a-b643-213236e27cde-kube-api-access-w8lhd\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.349570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.349523    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.349754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.349725    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.376960 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.376928    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:44.377169 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.376994    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:08:44.451669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.451630    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8lhd\" (UniqueName: \"kubernetes.io/projected/82f6b7ac-cd4e-404a-b643-213236e27cde-kube-api-access-w8lhd\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.451858 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.451731    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.451858 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.451828    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.452565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.452533    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.454373 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.454354    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.505287 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.505255    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8lhd\" (UniqueName: \"kubernetes.io/projected/82f6b7ac-cd4e-404a-b643-213236e27cde-kube-api-access-w8lhd\") pod \"maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.599063 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.599024    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:44.981815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:44.981733    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"]
Nov 03 21:08:44.986907 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:08:44.986871    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f6b7ac_cd4e_404a_b643_213236e27cde.slice/crio-d6fb3da268f421b8c47acc7b206419af28d5ba2008ad77c05969a69db998aef4 WatchSource:0}: Error finding container d6fb3da268f421b8c47acc7b206419af28d5ba2008ad77c05969a69db998aef4: Status 404 returned error can't find the container with id d6fb3da268f421b8c47acc7b206419af28d5ba2008ad77c05969a69db998aef4
Nov 03 21:08:45.187346 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:45.187303    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8" event={"ID":"82f6b7ac-cd4e-404a-b643-213236e27cde","Type":"ContainerStarted","Data":"d6fb3da268f421b8c47acc7b206419af28d5ba2008ad77c05969a69db998aef4"}
Nov 03 21:08:45.407634 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:45.407596    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"]
Nov 03 21:08:45.519391 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:45.519354    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"]
Nov 03 21:08:45.780985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:45.780891    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:45.925804 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:45.925773    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:46.198858 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:46.198817    5196 generic.go:356] "Generic (PLEG): container finished" podID="82f6b7ac-cd4e-404a-b643-213236e27cde" containerID="7810d5208786999cdd53136e2a55fe50557167fe37c7338a1d06d64043a61028" exitCode=0
Nov 03 21:08:46.199054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:46.198869    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8" event={"ID":"82f6b7ac-cd4e-404a-b643-213236e27cde","Type":"ContainerDied","Data":"7810d5208786999cdd53136e2a55fe50557167fe37c7338a1d06d64043a61028"}
Nov 03 21:08:46.540270 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:46.540178    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:47.495085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.495060    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:47.521944 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.521856    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f6b7ac-cd4e-404a-b643-213236e27cde-kube-api-access-w8lhd\" (UniqueName: \"kubernetes.io/projected/82f6b7ac-cd4e-404a-b643-213236e27cde-kube-api-access-w8lhd\") pod \"82f6b7ac-cd4e-404a-b643-213236e27cde\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") "
Nov 03 21:08:47.522161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.522010    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-openshift-ca\") pod \"82f6b7ac-cd4e-404a-b643-213236e27cde\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") "
Nov 03 21:08:47.522161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.522138    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-xks-ca\") pod \"82f6b7ac-cd4e-404a-b643-213236e27cde\" (UID: \"82f6b7ac-cd4e-404a-b643-213236e27cde\") "
Nov 03 21:08:47.522502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.522468    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-openshift-ca" pod "82f6b7ac-cd4e-404a-b643-213236e27cde" (UID: "82f6b7ac-cd4e-404a-b643-213236e27cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:47.522903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.522885    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:47.524398 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.524366    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-xks-ca" pod "82f6b7ac-cd4e-404a-b643-213236e27cde" (UID: "82f6b7ac-cd4e-404a-b643-213236e27cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:47.524500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.524464    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f6b7ac-cd4e-404a-b643-213236e27cde-kube-api-access-w8lhd" pod "82f6b7ac-cd4e-404a-b643-213236e27cde" (UID: "82f6b7ac-cd4e-404a-b643-213236e27cde"). InnerVolumeSpecName "kube-api-access-w8lhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:47.624177 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.624131    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8lhd\" (UniqueName: \"kubernetes.io/projected/82f6b7ac-cd4e-404a-b643-213236e27cde-kube-api-access-w8lhd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:47.624177 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:47.624172    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f6b7ac-cd4e-404a-b643-213236e27cde-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:48.260629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:48.260595    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"
Nov 03 21:08:48.260629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:48.260620    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8" event={"ID":"82f6b7ac-cd4e-404a-b643-213236e27cde","Type":"ContainerDied","Data":"d6fb3da268f421b8c47acc7b206419af28d5ba2008ad77c05969a69db998aef4"}
Nov 03 21:08:48.260852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:48.260675    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6fb3da268f421b8c47acc7b206419af28d5ba2008ad77c05969a69db998aef4"
Nov 03 21:08:48.486954 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:08:48.486857    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:08:49.102559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:49.102520    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:49.323711 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:49.323675    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:50.408440 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.408396    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p" podUID="9e990412-4a9b-43dc-9b2d-dea49496ce3f" containerName="payload-processing" containerID="cri-o://765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b" gracePeriod=25
Nov 03 21:08:50.520617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.520568    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr" podUID="5ffeef1b-d187-4e98-a791-d47cc5155c8e" containerName="payload-pre-processing" containerID="cri-o://98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac" gracePeriod=25
Nov 03 21:08:50.725136 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.725105    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:08:50.772854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.772821    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e990412-4a9b-43dc-9b2d-dea49496ce3f-kube-api-access-s5g5w\" (UniqueName: \"kubernetes.io/projected/9e990412-4a9b-43dc-9b2d-dea49496ce3f-kube-api-access-s5g5w\") pod \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\" (UID: \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\") "
Nov 03 21:08:50.773148 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.772967    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e990412-4a9b-43dc-9b2d-dea49496ce3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e990412-4a9b-43dc-9b2d-dea49496ce3f-plugins-config-volume\") pod \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\" (UID: \"9e990412-4a9b-43dc-9b2d-dea49496ce3f\") "
Nov 03 21:08:50.773451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.773419    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e990412-4a9b-43dc-9b2d-dea49496ce3f-plugins-config-volume" pod "9e990412-4a9b-43dc-9b2d-dea49496ce3f" (UID: "9e990412-4a9b-43dc-9b2d-dea49496ce3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:50.774105 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.774059    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e990412-4a9b-43dc-9b2d-dea49496ce3f-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:50.781348 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.781313    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e990412-4a9b-43dc-9b2d-dea49496ce3f-kube-api-access-s5g5w" pod "9e990412-4a9b-43dc-9b2d-dea49496ce3f" (UID: "9e990412-4a9b-43dc-9b2d-dea49496ce3f"). InnerVolumeSpecName "kube-api-access-s5g5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:50.805141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.805113    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:08:50.874853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.874814    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffeef1b-d187-4e98-a791-d47cc5155c8e-kube-api-access-q2cq4\" (UniqueName: \"kubernetes.io/projected/5ffeef1b-d187-4e98-a791-d47cc5155c8e-kube-api-access-q2cq4\") pod \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\" (UID: \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\") "
Nov 03 21:08:50.875066 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.875010    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ffeef1b-d187-4e98-a791-d47cc5155c8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffeef1b-d187-4e98-a791-d47cc5155c8e-plugins-config-volume\") pod \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\" (UID: \"5ffeef1b-d187-4e98-a791-d47cc5155c8e\") "
Nov 03 21:08:50.875551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.875498    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffeef1b-d187-4e98-a791-d47cc5155c8e-plugins-config-volume" pod "5ffeef1b-d187-4e98-a791-d47cc5155c8e" (UID: "5ffeef1b-d187-4e98-a791-d47cc5155c8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:50.875734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.875708    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5g5w\" (UniqueName: \"kubernetes.io/projected/9e990412-4a9b-43dc-9b2d-dea49496ce3f-kube-api-access-s5g5w\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:50.875868 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.875737    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffeef1b-d187-4e98-a791-d47cc5155c8e-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:50.877874 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.877845    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffeef1b-d187-4e98-a791-d47cc5155c8e-kube-api-access-q2cq4" pod "5ffeef1b-d187-4e98-a791-d47cc5155c8e" (UID: "5ffeef1b-d187-4e98-a791-d47cc5155c8e"). InnerVolumeSpecName "kube-api-access-q2cq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:50.977135 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:50.977032    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2cq4\" (UniqueName: \"kubernetes.io/projected/5ffeef1b-d187-4e98-a791-d47cc5155c8e-kube-api-access-q2cq4\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.288715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.288628    5196 generic.go:356] "Generic (PLEG): container finished" podID="9e990412-4a9b-43dc-9b2d-dea49496ce3f" containerID="765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b" exitCode=0
Nov 03 21:08:51.288715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.288681    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"
Nov 03 21:08:51.288982 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.288714    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p" event={"ID":"9e990412-4a9b-43dc-9b2d-dea49496ce3f","Type":"ContainerDied","Data":"765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b"}
Nov 03 21:08:51.288982 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.288774    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p" event={"ID":"9e990412-4a9b-43dc-9b2d-dea49496ce3f","Type":"ContainerDied","Data":"e94da3df2009fb6007e0034591e1ef5e39bb6a6c23492ae1f1dcc8b7ebc57bdc"}
Nov 03 21:08:51.288982 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.288795    5196 scope.go:122] "RemoveContainer" containerID="765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b"
Nov 03 21:08:51.290759 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.290735    5196 generic.go:356] "Generic (PLEG): container finished" podID="5ffeef1b-d187-4e98-a791-d47cc5155c8e" containerID="98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac" exitCode=0
Nov 03 21:08:51.290907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.290786    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr" event={"ID":"5ffeef1b-d187-4e98-a791-d47cc5155c8e","Type":"ContainerDied","Data":"98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac"}
Nov 03 21:08:51.290907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.290793    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"
Nov 03 21:08:51.290907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.290828    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr" event={"ID":"5ffeef1b-d187-4e98-a791-d47cc5155c8e","Type":"ContainerDied","Data":"48b79dcf22e799ab441c75ed5dccd8981db8753ffcf4bd16e0f3851ee4e363d0"}
Nov 03 21:08:51.302792 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.302769    5196 scope.go:122] "RemoveContainer" containerID="765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b"
Nov 03 21:08:51.303265 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:08:51.303239    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b\": container with ID starting with 765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b not found: ID does not exist" containerID="765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b"
Nov 03 21:08:51.303386 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.303276    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b"} err="failed to get container status \"765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b\": rpc error: code = NotFound desc = could not find container \"765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b\": container with ID starting with 765d01368bac478d1fe2059f8cb63ad591875db879750152e450afb30c31b16b not found: ID does not exist"
Nov 03 21:08:51.303386 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.303301    5196 scope.go:122] "RemoveContainer" containerID="98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac"
Nov 03 21:08:51.315738 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.315717    5196 scope.go:122] "RemoveContainer" containerID="98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac"
Nov 03 21:08:51.316067 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:08:51.316046    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac\": container with ID starting with 98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac not found: ID does not exist" containerID="98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac"
Nov 03 21:08:51.316226 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.316079    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac"} err="failed to get container status \"98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac\": rpc error: code = NotFound desc = could not find container \"98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac\": container with ID starting with 98fffde9e0b9ac629f678adaa1de8594b5502d9d94a4293b9025b9499b8f9aac not found: ID does not exist"
Nov 03 21:08:51.444817 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.444770    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"]
Nov 03 21:08:51.454295 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.454253    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c571ef-cb59656dc-hcbjr"]
Nov 03 21:08:51.549042 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.548950    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"]
Nov 03 21:08:51.553480 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.553449    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c571ef-566d45f799-w9x4p"]
Nov 03 21:08:51.865359 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:51.865323    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:52.089953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.089919    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:52.110900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.110869    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ffeef1b-d187-4e98-a791-d47cc5155c8e" path="/var/lib/kubelet/pods/5ffeef1b-d187-4e98-a791-d47cc5155c8e/volumes"
Nov 03 21:08:52.111432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.111416    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e990412-4a9b-43dc-9b2d-dea49496ce3f" path="/var/lib/kubelet/pods/9e990412-4a9b-43dc-9b2d-dea49496ce3f/volumes"
Nov 03 21:08:52.543780 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.543737    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"]
Nov 03 21:08:52.545023 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.544998    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5ffeef1b-d187-4e98-a791-d47cc5155c8e" containerName="payload-pre-processing"
Nov 03 21:08:52.545083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.545027    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffeef1b-d187-4e98-a791-d47cc5155c8e" containerName="payload-pre-processing"
Nov 03 21:08:52.545083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.545075    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9e990412-4a9b-43dc-9b2d-dea49496ce3f" containerName="payload-processing"
Nov 03 21:08:52.545194 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.545085    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e990412-4a9b-43dc-9b2d-dea49496ce3f" containerName="payload-processing"
Nov 03 21:08:52.545194 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.545151    5196 cpu_manager.go:414] "containerMap: removing container" podUID="82f6b7ac-cd4e-404a-b643-213236e27cde" containerName="revoke-keys"
Nov 03 21:08:52.545194 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.545160    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f6b7ac-cd4e-404a-b643-213236e27cde" containerName="revoke-keys"
Nov 03 21:08:52.545414 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.545397    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffeef1b-d187-4e98-a791-d47cc5155c8e" containerName="payload-pre-processing"
Nov 03 21:08:52.545453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.545426    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e990412-4a9b-43dc-9b2d-dea49496ce3f" containerName="payload-processing"
Nov 03 21:08:52.545453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.545439    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f6b7ac-cd4e-404a-b643-213236e27cde" containerName="revoke-keys"
Nov 03 21:08:52.551432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.551407    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.599436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.599406    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"]
Nov 03 21:08:52.639687 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.639649    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:52.639908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.639752    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:08:52.704702 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.704671    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.704923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.704713    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.704923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.704885    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngrg\" (UniqueName: \"kubernetes.io/projected/01c2044d-b1a0-48fd-8299-e47a349d16b2-kube-api-access-zngrg\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.806048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.805961    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.806048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.806009    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.806312 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.806137    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngrg\" (UniqueName: \"kubernetes.io/projected/01c2044d-b1a0-48fd-8299-e47a349d16b2-kube-api-access-zngrg\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.806631 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.806611    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.808616 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.808597    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.860073 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.860035    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngrg\" (UniqueName: \"kubernetes.io/projected/01c2044d-b1a0-48fd-8299-e47a349d16b2-kube-api-access-zngrg\") pod \"maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:52.866804 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:52.866775    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:53.237982 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:53.237941    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:53.253571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:53.253505    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"]
Nov 03 21:08:53.259410 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:08:53.259376    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c2044d_b1a0_48fd_8299_e47a349d16b2.slice/crio-687f733b47d834a2beb80d62ccb6e2cca977a856a159ac99414285cff8da692a WatchSource:0}: Error finding container 687f733b47d834a2beb80d62ccb6e2cca977a856a159ac99414285cff8da692a: Status 404 returned error can't find the container with id 687f733b47d834a2beb80d62ccb6e2cca977a856a159ac99414285cff8da692a
Nov 03 21:08:53.315449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:53.315414    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2" event={"ID":"01c2044d-b1a0-48fd-8299-e47a349d16b2","Type":"ContainerStarted","Data":"687f733b47d834a2beb80d62ccb6e2cca977a856a159ac99414285cff8da692a"}
Nov 03 21:08:54.326835 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:54.326796    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2" event={"ID":"01c2044d-b1a0-48fd-8299-e47a349d16b2","Type":"ContainerStarted","Data":"ce55b61852b879e8479f032ec646132dae186a3c8b402f5265093f706de963bf"}
Nov 03 21:08:54.436671 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:54.436615    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2" podStartSLOduration=2.436601248 podStartE2EDuration="2.436601248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:54.433576153 +0000 UTC m=+2374.437534888" watchObservedRunningTime="2026-11-03 21:08:54.436601248 +0000 UTC m=+2374.440560014"
Nov 03 21:08:55.337645 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:55.337611    5196 generic.go:356] "Generic (PLEG): container finished" podID="01c2044d-b1a0-48fd-8299-e47a349d16b2" containerID="ce55b61852b879e8479f032ec646132dae186a3c8b402f5265093f706de963bf" exitCode=0
Nov 03 21:08:55.338080 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:55.337656    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2" event={"ID":"01c2044d-b1a0-48fd-8299-e47a349d16b2","Type":"ContainerDied","Data":"ce55b61852b879e8479f032ec646132dae186a3c8b402f5265093f706de963bf"}
Nov 03 21:08:56.439296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.439181    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:56.631475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.631356    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:56.632235 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.632208    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:56.767939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.767899    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-openshift-ca\") pod \"01c2044d-b1a0-48fd-8299-e47a349d16b2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") "
Nov 03 21:08:56.768168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.767963    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-xks-ca\") pod \"01c2044d-b1a0-48fd-8299-e47a349d16b2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") "
Nov 03 21:08:56.768168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.767994    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c2044d-b1a0-48fd-8299-e47a349d16b2-kube-api-access-zngrg\" (UniqueName: \"kubernetes.io/projected/01c2044d-b1a0-48fd-8299-e47a349d16b2-kube-api-access-zngrg\") pod \"01c2044d-b1a0-48fd-8299-e47a349d16b2\" (UID: \"01c2044d-b1a0-48fd-8299-e47a349d16b2\") "
Nov 03 21:08:56.768477 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.768448    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-openshift-ca" pod "01c2044d-b1a0-48fd-8299-e47a349d16b2" (UID: "01c2044d-b1a0-48fd-8299-e47a349d16b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:56.769006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.768979    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:56.771177 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.771150    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-xks-ca" pod "01c2044d-b1a0-48fd-8299-e47a349d16b2" (UID: "01c2044d-b1a0-48fd-8299-e47a349d16b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:56.771304 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.771276    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c2044d-b1a0-48fd-8299-e47a349d16b2-kube-api-access-zngrg" pod "01c2044d-b1a0-48fd-8299-e47a349d16b2" (UID: "01c2044d-b1a0-48fd-8299-e47a349d16b2"). InnerVolumeSpecName "kube-api-access-zngrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:56.870291 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.870262    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c2044d-b1a0-48fd-8299-e47a349d16b2-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:56.870468 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:56.870288    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngrg\" (UniqueName: \"kubernetes.io/projected/01c2044d-b1a0-48fd-8299-e47a349d16b2-kube-api-access-zngrg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:57.118498 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:57.118460    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:57.312974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:57.312939    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:57.358973 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:57.358938    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"
Nov 03 21:08:57.358973 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:57.358955    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2" event={"ID":"01c2044d-b1a0-48fd-8299-e47a349d16b2","Type":"ContainerDied","Data":"687f733b47d834a2beb80d62ccb6e2cca977a856a159ac99414285cff8da692a"}
Nov 03 21:08:57.359220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:08:57.359038    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="687f733b47d834a2beb80d62ccb6e2cca977a856a159ac99414285cff8da692a"
Nov 03 21:09:02.082162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:02.082129    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:02.729667 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:02.729629    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:09:03.327183 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:03.327143    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"]
Nov 03 21:09:03.440859 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:03.440824    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"]
Nov 03 21:09:03.700797 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:03.700708    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:03.848458 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:03.848422    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:05.540841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:05.540794    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:06.010464 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.010424    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"]
Nov 03 21:09:06.011390 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.011372    5196 cpu_manager.go:414] "containerMap: removing container" podUID="01c2044d-b1a0-48fd-8299-e47a349d16b2" containerName="revoke-keys"
Nov 03 21:09:06.011453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.011392    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c2044d-b1a0-48fd-8299-e47a349d16b2" containerName="revoke-keys"
Nov 03 21:09:06.011528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.011518    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c2044d-b1a0-48fd-8299-e47a349d16b2" containerName="revoke-keys"
Nov 03 21:09:06.017187 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.017164    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.053761 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.053722    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.053981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.053909    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4sr\" (UniqueName: \"kubernetes.io/projected/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-kube-api-access-hc4sr\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.054050 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.054031    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.067027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.066996    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"]
Nov 03 21:09:06.108059 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.108033    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:06.108281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.108188    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:09:06.155753 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.155711    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.155946 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.155811    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4sr\" (UniqueName: \"kubernetes.io/projected/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-kube-api-access-hc4sr\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.155946 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.155882    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.156504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.156479    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.158593 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.158565    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.210163 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.210113    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4sr\" (UniqueName: \"kubernetes.io/projected/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-kube-api-access-hc4sr\") pod \"maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.332921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.332891    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:06.547079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.547038    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"]
Nov 03 21:09:06.553168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.553140    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.561421 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.561385    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.561673 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.561651    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25cvq\" (UniqueName: \"kubernetes.io/projected/ef28e282-ce5f-479d-a403-7ea65ef54d24-kube-api-access-25cvq\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.561790 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.561768    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.607548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.606842    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"]
Nov 03 21:09:06.618075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.618041    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"]
Nov 03 21:09:06.618256 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.618195    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.664664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.664628    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"]
Nov 03 21:09:06.664949 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.664752    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.665064 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.664792    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.665197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.665139    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.665276 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.665213    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.665562 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.665407    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q6z7\" (UniqueName: \"kubernetes.io/projected/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-kube-api-access-7q6z7\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.665562 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.665448    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25cvq\" (UniqueName: \"kubernetes.io/projected/ef28e282-ce5f-479d-a403-7ea65ef54d24-kube-api-access-25cvq\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.665562 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.665493    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.668820 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.668790    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.720087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.720052    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"]
Nov 03 21:09:06.720523 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.720494    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25cvq\" (UniqueName: \"kubernetes.io/projected/ef28e282-ce5f-479d-a403-7ea65ef54d24-kube-api-access-25cvq\") pod \"maas-api-revoke-sub-14694eb2-g8cz2\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.725661 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:09:06.725634    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221afa29_4ac5_4bed_8e2b_bc2eb593ac56.slice/crio-dcda898f2e8ff1353fdf52e26aacbcf63ef559d8547ef5ca98a47b8fcf5dbdc6 WatchSource:0}: Error finding container dcda898f2e8ff1353fdf52e26aacbcf63ef559d8547ef5ca98a47b8fcf5dbdc6: Status 404 returned error can't find the container with id dcda898f2e8ff1353fdf52e26aacbcf63ef559d8547ef5ca98a47b8fcf5dbdc6
Nov 03 21:09:06.767316 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.767284    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.767498 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.767403    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q6z7\" (UniqueName: \"kubernetes.io/projected/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-kube-api-access-7q6z7\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.767498 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.767438    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.768070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.768037    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.770389 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.770360    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.822245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.822204    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q6z7\" (UniqueName: \"kubernetes.io/projected/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-kube-api-access-7q6z7\") pod \"maas-api-revoke-sub-9e8f4f22-k4ctz\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:06.869509 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.869414    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:06.937837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:06.937810    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:07.318729 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:07.318703    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"]
Nov 03 21:09:07.321646 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:09:07.321616    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef28e282_ce5f_479d_a403_7ea65ef54d24.slice/crio-24a9bba61b0acd33084c6c25d2b1795a9130580194f731aa873f11b8b30b3b95 WatchSource:0}: Error finding container 24a9bba61b0acd33084c6c25d2b1795a9130580194f731aa873f11b8b30b3b95: Status 404 returned error can't find the container with id 24a9bba61b0acd33084c6c25d2b1795a9130580194f731aa873f11b8b30b3b95
Nov 03 21:09:07.385114 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:07.383316    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"]
Nov 03 21:09:07.389568 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:09:07.389525    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b00d510_0f2b_4896_8cf2_0bcb280bcdb8.slice/crio-654af286775538afd6bbb06492334ce7de4e42251c901bce6b7be909e6372c19 WatchSource:0}: Error finding container 654af286775538afd6bbb06492334ce7de4e42251c901bce6b7be909e6372c19: Status 404 returned error can't find the container with id 654af286775538afd6bbb06492334ce7de4e42251c901bce6b7be909e6372c19
Nov 03 21:09:07.451468 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:07.451427    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz" event={"ID":"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8","Type":"ContainerStarted","Data":"654af286775538afd6bbb06492334ce7de4e42251c901bce6b7be909e6372c19"}
Nov 03 21:09:07.452896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:07.452867    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2" event={"ID":"ef28e282-ce5f-479d-a403-7ea65ef54d24","Type":"ContainerStarted","Data":"24a9bba61b0acd33084c6c25d2b1795a9130580194f731aa873f11b8b30b3b95"}
Nov 03 21:09:07.454840 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:07.454802    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z" event={"ID":"221afa29-4ac5-4bed-8e2b-bc2eb593ac56","Type":"ContainerStarted","Data":"04a485c41af00093f2c967f125e3d4f6a2e5ebe6e395f79e6de42674f669eb8a"}
Nov 03 21:09:07.455010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:07.454861    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z" event={"ID":"221afa29-4ac5-4bed-8e2b-bc2eb593ac56","Type":"ContainerStarted","Data":"dcda898f2e8ff1353fdf52e26aacbcf63ef559d8547ef5ca98a47b8fcf5dbdc6"}
Nov 03 21:09:07.456765 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:07.456729    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:07.570656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:07.570598    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z" podStartSLOduration=2.570580773 podStartE2EDuration="2.570580773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:07.566404668 +0000 UTC m=+2387.570363392" watchObservedRunningTime="2026-11-03 21:09:07.570580773 +0000 UTC m=+2387.574539496"
Nov 03 21:09:08.327884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.327839    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv" podUID="ccbfd999-a314-42bd-9c1d-b8bcd1f7e699" containerName="payload-processing" containerID="cri-o://9a782d2974c7564446c495d7142eb96145768ff315655f4b04d0d8766cb32aeb" gracePeriod=25
Nov 03 21:09:08.441524 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.441404    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5" podUID="3a757404-5bf1-4e94-ba40-2c8ce0d83959" containerName="payload-pre-processing" containerID="cri-o://558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a" gracePeriod=25
Nov 03 21:09:08.467810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.467771    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz" event={"ID":"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8","Type":"ContainerStarted","Data":"5c06e89f12be429e7d91808f865e585d4a1b97527104611bf090713c1473ca09"}
Nov 03 21:09:08.470639 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.470518    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2" event={"ID":"ef28e282-ce5f-479d-a403-7ea65ef54d24","Type":"ContainerStarted","Data":"fc0a3667b4d40d26297be480944e9203cf1be5d9bcd8b511d71d9567be39d960"}
Nov 03 21:09:08.472998 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.472968    5196 generic.go:356] "Generic (PLEG): container finished" podID="ccbfd999-a314-42bd-9c1d-b8bcd1f7e699" containerID="9a782d2974c7564446c495d7142eb96145768ff315655f4b04d0d8766cb32aeb" exitCode=0
Nov 03 21:09:08.473178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.473043    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv" event={"ID":"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699","Type":"ContainerDied","Data":"9a782d2974c7564446c495d7142eb96145768ff315655f4b04d0d8766cb32aeb"}
Nov 03 21:09:08.474926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.474898    5196 generic.go:356] "Generic (PLEG): container finished" podID="221afa29-4ac5-4bed-8e2b-bc2eb593ac56" containerID="04a485c41af00093f2c967f125e3d4f6a2e5ebe6e395f79e6de42674f669eb8a" exitCode=0
Nov 03 21:09:08.475042 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.474978    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z" event={"ID":"221afa29-4ac5-4bed-8e2b-bc2eb593ac56","Type":"ContainerDied","Data":"04a485c41af00093f2c967f125e3d4f6a2e5ebe6e395f79e6de42674f669eb8a"}
Nov 03 21:09:08.578892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.578820    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz" podStartSLOduration=2.578799854 podStartE2EDuration="2.578799854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:08.577150357 +0000 UTC m=+2388.581109108" watchObservedRunningTime="2026-11-03 21:09:08.578799854 +0000 UTC m=+2388.582758580"
Nov 03 21:09:08.655972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.655943    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 21:09:08.696770 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.696671    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-kube-api-access-fqmps\" (UniqueName: \"kubernetes.io/projected/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-kube-api-access-fqmps\") pod \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\" (UID: \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\") "
Nov 03 21:09:08.696770 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.696734    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-plugins-config-volume\") pod \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\" (UID: \"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699\") "
Nov 03 21:09:08.697240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.697200    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-plugins-config-volume" pod "ccbfd999-a314-42bd-9c1d-b8bcd1f7e699" (UID: "ccbfd999-a314-42bd-9c1d-b8bcd1f7e699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:08.697514 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.697490    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:08.700319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.700283    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-kube-api-access-fqmps" pod "ccbfd999-a314-42bd-9c1d-b8bcd1f7e699" (UID: "ccbfd999-a314-42bd-9c1d-b8bcd1f7e699"). InnerVolumeSpecName "kube-api-access-fqmps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:08.744362 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.744335    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 21:09:08.798870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.798826    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a757404-5bf1-4e94-ba40-2c8ce0d83959-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757404-5bf1-4e94-ba40-2c8ce0d83959-plugins-config-volume\") pod \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\" (UID: \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\") "
Nov 03 21:09:08.799158 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.799139    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a757404-5bf1-4e94-ba40-2c8ce0d83959-kube-api-access-7vh79\" (UniqueName: \"kubernetes.io/projected/3a757404-5bf1-4e94-ba40-2c8ce0d83959-kube-api-access-7vh79\") pod \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\" (UID: \"3a757404-5bf1-4e94-ba40-2c8ce0d83959\") "
Nov 03 21:09:08.799229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.799204    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a757404-5bf1-4e94-ba40-2c8ce0d83959-plugins-config-volume" pod "3a757404-5bf1-4e94-ba40-2c8ce0d83959" (UID: "3a757404-5bf1-4e94-ba40-2c8ce0d83959"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:08.799868 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.799783    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqmps\" (UniqueName: \"kubernetes.io/projected/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699-kube-api-access-fqmps\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:08.799868 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.799817    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757404-5bf1-4e94-ba40-2c8ce0d83959-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:08.801787 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.801759    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a757404-5bf1-4e94-ba40-2c8ce0d83959-kube-api-access-7vh79" pod "3a757404-5bf1-4e94-ba40-2c8ce0d83959" (UID: "3a757404-5bf1-4e94-ba40-2c8ce0d83959"). InnerVolumeSpecName "kube-api-access-7vh79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:08.801787 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.801745    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2" podStartSLOduration=2.801732494 podStartE2EDuration="2.801732494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:08.79602958 +0000 UTC m=+2388.799988341" watchObservedRunningTime="2026-11-03 21:09:08.801732494 +0000 UTC m=+2388.805691217"
Nov 03 21:09:08.901341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:08.901299    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vh79\" (UniqueName: \"kubernetes.io/projected/3a757404-5bf1-4e94-ba40-2c8ce0d83959-kube-api-access-7vh79\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:09.485350 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.485260    5196 generic.go:356] "Generic (PLEG): container finished" podID="3a757404-5bf1-4e94-ba40-2c8ce0d83959" containerID="558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a" exitCode=0
Nov 03 21:09:09.485350 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.485327    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"
Nov 03 21:09:09.485589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.485319    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5" event={"ID":"3a757404-5bf1-4e94-ba40-2c8ce0d83959","Type":"ContainerDied","Data":"558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a"}
Nov 03 21:09:09.485589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.485454    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5" event={"ID":"3a757404-5bf1-4e94-ba40-2c8ce0d83959","Type":"ContainerDied","Data":"8a687c67d1f37dc4bd651119d4e57bdfd22e81d89641a38770de80f38dc9a65b"}
Nov 03 21:09:09.485589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.485482    5196 scope.go:122] "RemoveContainer" containerID="558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a"
Nov 03 21:09:09.487604 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.487577    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv" event={"ID":"ccbfd999-a314-42bd-9c1d-b8bcd1f7e699","Type":"ContainerDied","Data":"e7f405e10e8bc64cffea6cd9e660c0d6ac2cbc0a8e9024b58bbd0e5c91d4d993"}
Nov 03 21:09:09.487739 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.487610    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"
Nov 03 21:09:09.489844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.489812    5196 generic.go:356] "Generic (PLEG): container finished" podID="4b00d510-0f2b-4896-8cf2-0bcb280bcdb8" containerID="5c06e89f12be429e7d91808f865e585d4a1b97527104611bf090713c1473ca09" exitCode=0
Nov 03 21:09:09.489959 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.489844    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz" event={"ID":"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8","Type":"ContainerDied","Data":"5c06e89f12be429e7d91808f865e585d4a1b97527104611bf090713c1473ca09"}
Nov 03 21:09:09.496964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.496938    5196 generic.go:356] "Generic (PLEG): container finished" podID="ef28e282-ce5f-479d-a403-7ea65ef54d24" containerID="fc0a3667b4d40d26297be480944e9203cf1be5d9bcd8b511d71d9567be39d960" exitCode=0
Nov 03 21:09:09.497141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.496994    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2" event={"ID":"ef28e282-ce5f-479d-a403-7ea65ef54d24","Type":"ContainerDied","Data":"fc0a3667b4d40d26297be480944e9203cf1be5d9bcd8b511d71d9567be39d960"}
Nov 03 21:09:09.499444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.499426    5196 scope.go:122] "RemoveContainer" containerID="558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a"
Nov 03 21:09:09.499725 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:09:09.499703    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a\": container with ID starting with 558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a not found: ID does not exist" containerID="558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a"
Nov 03 21:09:09.499771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.499735    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a"} err="failed to get container status \"558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a\": rpc error: code = NotFound desc = could not find container \"558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a\": container with ID starting with 558f607c97b98c6465edfa7d2b0e7ad2e150f1e04f01bd8ca62f6c63fd35a97a not found: ID does not exist"
Nov 03 21:09:09.499771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.499753    5196 scope.go:122] "RemoveContainer" containerID="9a782d2974c7564446c495d7142eb96145768ff315655f4b04d0d8766cb32aeb"
Nov 03 21:09:09.678193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.678164    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:09.715599 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.715554    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-openshift-ca\") pod \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") "
Nov 03 21:09:09.715821 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.715680    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-xks-ca\") pod \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") "
Nov 03 21:09:09.715894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.715879    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-kube-api-access-hc4sr\" (UniqueName: \"kubernetes.io/projected/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-kube-api-access-hc4sr\") pod \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\" (UID: \"221afa29-4ac5-4bed-8e2b-bc2eb593ac56\") "
Nov 03 21:09:09.716014 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.715992    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-openshift-ca" pod "221afa29-4ac5-4bed-8e2b-bc2eb593ac56" (UID: "221afa29-4ac5-4bed-8e2b-bc2eb593ac56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:09.716760 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.716632    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:09.718281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.718257    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-xks-ca" pod "221afa29-4ac5-4bed-8e2b-bc2eb593ac56" (UID: "221afa29-4ac5-4bed-8e2b-bc2eb593ac56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:09.718375 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.718338    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-kube-api-access-hc4sr" pod "221afa29-4ac5-4bed-8e2b-bc2eb593ac56" (UID: "221afa29-4ac5-4bed-8e2b-bc2eb593ac56"). InnerVolumeSpecName "kube-api-access-hc4sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:09.749804 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.749728    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"]
Nov 03 21:09:09.754441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.754414    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a19980-6cb7cbb6d7-g5bcv"]
Nov 03 21:09:09.818323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.818280    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4sr\" (UniqueName: \"kubernetes.io/projected/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-kube-api-access-hc4sr\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:09.818323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.818318    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221afa29-4ac5-4bed-8e2b-bc2eb593ac56-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:09.963299 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.963263    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"]
Nov 03 21:09:09.968785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:09.968754    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a19980-67fcb6d95-l9bm5"]
Nov 03 21:09:10.119610 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.119573    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a757404-5bf1-4e94-ba40-2c8ce0d83959" path="/var/lib/kubelet/pods/3a757404-5bf1-4e94-ba40-2c8ce0d83959/volumes"
Nov 03 21:09:10.120483 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.120460    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbfd999-a314-42bd-9c1d-b8bcd1f7e699" path="/var/lib/kubelet/pods/ccbfd999-a314-42bd-9c1d-b8bcd1f7e699/volumes"
Nov 03 21:09:10.484633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.484529    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"]
Nov 03 21:09:10.511016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.510983    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"
Nov 03 21:09:10.511016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.511009    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z" event={"ID":"221afa29-4ac5-4bed-8e2b-bc2eb593ac56","Type":"ContainerDied","Data":"dcda898f2e8ff1353fdf52e26aacbcf63ef559d8547ef5ca98a47b8fcf5dbdc6"}
Nov 03 21:09:10.511289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.511061    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcda898f2e8ff1353fdf52e26aacbcf63ef559d8547ef5ca98a47b8fcf5dbdc6"
Nov 03 21:09:10.532051 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.532010    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"]
Nov 03 21:09:10.681991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.681440    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:10.716902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.716863    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:10.764574 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.764499    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:10.832628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.832589    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-kube-api-access-7q6z7\" (UniqueName: \"kubernetes.io/projected/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-kube-api-access-7q6z7\") pod \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") "
Nov 03 21:09:10.832837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.832656    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-xks-ca\") pod \"ef28e282-ce5f-479d-a403-7ea65ef54d24\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") "
Nov 03 21:09:10.832837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.832676    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef28e282-ce5f-479d-a403-7ea65ef54d24-kube-api-access-25cvq\" (UniqueName: \"kubernetes.io/projected/ef28e282-ce5f-479d-a403-7ea65ef54d24-kube-api-access-25cvq\") pod \"ef28e282-ce5f-479d-a403-7ea65ef54d24\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") "
Nov 03 21:09:10.832837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.832715    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-openshift-ca\") pod \"ef28e282-ce5f-479d-a403-7ea65ef54d24\" (UID: \"ef28e282-ce5f-479d-a403-7ea65ef54d24\") "
Nov 03 21:09:10.832837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.832746    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-xks-ca\") pod \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") "
Nov 03 21:09:10.833197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.833157    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-openshift-ca\") pod \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\" (UID: \"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8\") "
Nov 03 21:09:10.833360 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.833160    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-openshift-ca" pod "ef28e282-ce5f-479d-a403-7ea65ef54d24" (UID: "ef28e282-ce5f-479d-a403-7ea65ef54d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:10.833506 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.833482    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-openshift-ca" pod "4b00d510-0f2b-4896-8cf2-0bcb280bcdb8" (UID: "4b00d510-0f2b-4896-8cf2-0bcb280bcdb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:10.833847 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.833828    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:10.833910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.833852    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:10.835810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.835775    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-xks-ca" pod "ef28e282-ce5f-479d-a403-7ea65ef54d24" (UID: "ef28e282-ce5f-479d-a403-7ea65ef54d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:10.835916 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.835853    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-xks-ca" pod "4b00d510-0f2b-4896-8cf2-0bcb280bcdb8" (UID: "4b00d510-0f2b-4896-8cf2-0bcb280bcdb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:10.835978 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.835927    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-kube-api-access-7q6z7" pod "4b00d510-0f2b-4896-8cf2-0bcb280bcdb8" (UID: "4b00d510-0f2b-4896-8cf2-0bcb280bcdb8"). InnerVolumeSpecName "kube-api-access-7q6z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:10.835978 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.835964    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef28e282-ce5f-479d-a403-7ea65ef54d24-kube-api-access-25cvq" pod "ef28e282-ce5f-479d-a403-7ea65ef54d24" (UID: "ef28e282-ce5f-479d-a403-7ea65ef54d24"). InnerVolumeSpecName "kube-api-access-25cvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:10.935104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.935066    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q6z7\" (UniqueName: \"kubernetes.io/projected/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-kube-api-access-7q6z7\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:10.935104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.935114    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28e282-ce5f-479d-a403-7ea65ef54d24-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:10.935330 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.935126    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25cvq\" (UniqueName: \"kubernetes.io/projected/ef28e282-ce5f-479d-a403-7ea65ef54d24-kube-api-access-25cvq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:10.935330 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:10.935135    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:11.521494 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:11.521463    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"
Nov 03 21:09:11.521681 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:11.521486    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz" event={"ID":"4b00d510-0f2b-4896-8cf2-0bcb280bcdb8","Type":"ContainerDied","Data":"654af286775538afd6bbb06492334ce7de4e42251c901bce6b7be909e6372c19"}
Nov 03 21:09:11.521681 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:11.521549    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="654af286775538afd6bbb06492334ce7de4e42251c901bce6b7be909e6372c19"
Nov 03 21:09:11.523074 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:11.523055    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"
Nov 03 21:09:11.523074 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:11.523084    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2" event={"ID":"ef28e282-ce5f-479d-a403-7ea65ef54d24","Type":"ContainerDied","Data":"24a9bba61b0acd33084c6c25d2b1795a9130580194f731aa873f11b8b30b3b95"}
Nov 03 21:09:11.523362 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:11.523178    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24a9bba61b0acd33084c6c25d2b1795a9130580194f731aa873f11b8b30b3b95"
Nov 03 21:09:12.556935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:12.556897    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:14.833881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:14.833845    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:14.996128 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:14.993629    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:16.048321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:16.048276    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"]
Nov 03 21:09:16.096313 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:16.096276    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c60b7dd-4wsll"]
Nov 03 21:09:16.113193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:16.113155    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f15403-bcd6-441c-ade1-a36c77d4c8f0" path="/var/lib/kubelet/pods/38f15403-bcd6-441c-ade1-a36c77d4c8f0/volumes"
Nov 03 21:09:16.611917 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:16.605346    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"]
Nov 03 21:09:16.723042 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:16.723006    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"]
Nov 03 21:09:16.985983 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:16.985880    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:17.236147 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:17.236037    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:17.468901 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:17.468865    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:18.525035 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:09:18.519503    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:09:20.585288 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:20.585262    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:09:20.591526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:20.591504    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:09:20.600499 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:20.600471    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:09:20.607420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:20.607398    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:09:21.606208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:21.606166    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9" podUID="6f838b11-fd4b-492c-bf4e-15a5a0bf4208" containerName="payload-processing" containerID="cri-o://d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8" gracePeriod=25
Nov 03 21:09:21.608454 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:21.608419    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:21.723367 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:21.723325    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f" podUID="d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" containerName="payload-pre-processing" containerID="cri-o://ab8034821ef8c5679dd17813ff9b0f274b69a2421ab6e034d361f02ff2da4932" gracePeriod=25
Nov 03 21:09:22.430553 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:22.430508    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:22.585861 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:22.585826    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:23.372051 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:23.371998    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:23.533499 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:23.533467    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:23.681953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:23.681867    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:23.882511 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:23.882470    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"]
Nov 03 21:09:23.930542 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:23.930512    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-211d68bf-59cvl"]
Nov 03 21:09:24.112455 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:24.112420    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01ddc62-1b47-4f82-a746-78f3eec64c6d" path="/var/lib/kubelet/pods/b01ddc62-1b47-4f82-a746-78f3eec64c6d/volumes"
Nov 03 21:09:24.287464 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:24.287425    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:25.508545 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:25.508486    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main" containerID="cri-o://9932ca54c21cb6ad197995a3d7c50b3353cad1c675024dd57bcf16f5379ac5e1" gracePeriod=45
Nov 03 21:09:25.560187 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:25.560150    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main" containerID="cri-o://1f9859b0a2e54dadb8dd05e73bea4bc6969084a35c79de1c7e6cb60009c41ce2" gracePeriod=45
Nov 03 21:09:25.677907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:25.677873    5196 generic.go:356] "Generic (PLEG): container finished" podID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerID="9932ca54c21cb6ad197995a3d7c50b3353cad1c675024dd57bcf16f5379ac5e1" exitCode=0
Nov 03 21:09:25.678035 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:25.677943    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" event={"ID":"9cdca4ca-b9ea-4576-bc2b-21231f397ec9","Type":"ContainerDied","Data":"9932ca54c21cb6ad197995a3d7c50b3353cad1c675024dd57bcf16f5379ac5e1"}
Nov 03 21:09:25.680145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:25.680123    5196 generic.go:356] "Generic (PLEG): container finished" podID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerID="1f9859b0a2e54dadb8dd05e73bea4bc6969084a35c79de1c7e6cb60009c41ce2" exitCode=0
Nov 03 21:09:25.680274 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:25.680218    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" event={"ID":"f4d80abf-0dc2-4ba7-9bc6-3692275e9287","Type":"ContainerDied","Data":"1f9859b0a2e54dadb8dd05e73bea4bc6969084a35c79de1c7e6cb60009c41ce2"}
Nov 03 21:09:25.871581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:25.871547    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 21:09:25.904815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:25.904791    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 21:09:26.014503 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014470    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-home\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-home\") pod \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") "
Nov 03 21:09:26.014707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014532    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-kube-api-access-rxtgb\" (UniqueName: \"kubernetes.io/projected/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-kube-api-access-rxtgb\") pod \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") "
Nov 03 21:09:26.014707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014602    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tls-certs\") pod \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") "
Nov 03 21:09:26.014707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014647    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-model-cache\") pod \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") "
Nov 03 21:09:26.014895 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014722    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-model-cache\") pod \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") "
Nov 03 21:09:26.014895 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014796    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-home" pod "f4d80abf-0dc2-4ba7-9bc6-3692275e9287" (UID: "f4d80abf-0dc2-4ba7-9bc6-3692275e9287"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:26.014895 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014803    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-kube-api-access-nrkhp\" (UniqueName: \"kubernetes.io/projected/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-kube-api-access-nrkhp\") pod \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") "
Nov 03 21:09:26.015052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014904    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tmp-dir\") pod \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") "
Nov 03 21:09:26.015052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014942    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-home\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-home\") pod \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") "
Nov 03 21:09:26.015052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.014980    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tmp-dir\") pod \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") "
Nov 03 21:09:26.015245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.015052    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-dshm\") pod \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") "
Nov 03 21:09:26.015245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.015204    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-dshm\") pod \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\" (UID: \"9cdca4ca-b9ea-4576-bc2b-21231f397ec9\") "
Nov 03 21:09:26.015385 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.015333    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tls-certs\") pod \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\" (UID: \"f4d80abf-0dc2-4ba7-9bc6-3692275e9287\") "
Nov 03 21:09:26.015688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.015659    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tmp-dir" pod "9cdca4ca-b9ea-4576-bc2b-21231f397ec9" (UID: "9cdca4ca-b9ea-4576-bc2b-21231f397ec9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:26.015962 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.015735    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-model-cache" pod "f4d80abf-0dc2-4ba7-9bc6-3692275e9287" (UID: "f4d80abf-0dc2-4ba7-9bc6-3692275e9287"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:26.015962 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.015748    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-model-cache" pod "9cdca4ca-b9ea-4576-bc2b-21231f397ec9" (UID: "9cdca4ca-b9ea-4576-bc2b-21231f397ec9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:26.016242 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.016218    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-home" pod "9cdca4ca-b9ea-4576-bc2b-21231f397ec9" (UID: "9cdca4ca-b9ea-4576-bc2b-21231f397ec9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:26.016487 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.016449    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.016487 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.016473    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.016619 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.016488    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.016619 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.016501    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.016619 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.016516    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.016801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.016693    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tmp-dir" pod "f4d80abf-0dc2-4ba7-9bc6-3692275e9287" (UID: "f4d80abf-0dc2-4ba7-9bc6-3692275e9287"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:26.017693 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.017661    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-kube-api-access-nrkhp" pod "f4d80abf-0dc2-4ba7-9bc6-3692275e9287" (UID: "f4d80abf-0dc2-4ba7-9bc6-3692275e9287"). InnerVolumeSpecName "kube-api-access-nrkhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:26.017805 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.017783    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-kube-api-access-rxtgb" pod "9cdca4ca-b9ea-4576-bc2b-21231f397ec9" (UID: "9cdca4ca-b9ea-4576-bc2b-21231f397ec9"). InnerVolumeSpecName "kube-api-access-rxtgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:26.018955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.018321    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-dshm" pod "f4d80abf-0dc2-4ba7-9bc6-3692275e9287" (UID: "f4d80abf-0dc2-4ba7-9bc6-3692275e9287"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:26.018955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.018876    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tls-certs" pod "f4d80abf-0dc2-4ba7-9bc6-3692275e9287" (UID: "f4d80abf-0dc2-4ba7-9bc6-3692275e9287"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:26.019332 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.019307    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tls-certs" pod "9cdca4ca-b9ea-4576-bc2b-21231f397ec9" (UID: "9cdca4ca-b9ea-4576-bc2b-21231f397ec9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:26.020510 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.020482    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-dshm" pod "9cdca4ca-b9ea-4576-bc2b-21231f397ec9" (UID: "9cdca4ca-b9ea-4576-bc2b-21231f397ec9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:26.119366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.119313    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrkhp\" (UniqueName: \"kubernetes.io/projected/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-kube-api-access-nrkhp\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.119366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.119362    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.119366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.119376    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.119700 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.119390    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.119700 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.119403    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d80abf-0dc2-4ba7-9bc6-3692275e9287-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.119700 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.119416    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxtgb\" (UniqueName: \"kubernetes.io/projected/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-kube-api-access-rxtgb\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.119700 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.119429    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdca4ca-b9ea-4576-bc2b-21231f397ec9-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:26.236834 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.236798    5196 scope.go:122] "RemoveContainer" containerID="1f9859b0a2e54dadb8dd05e73bea4bc6969084a35c79de1c7e6cb60009c41ce2"
Nov 03 21:09:26.248791 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.248767    5196 scope.go:122] "RemoveContainer" containerID="5e4b81e9b07aab53ada191281f8d643975b38fafdd167e47c490618e781b72e6"
Nov 03 21:09:26.259726 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.259703    5196 scope.go:122] "RemoveContainer" containerID="9932ca54c21cb6ad197995a3d7c50b3353cad1c675024dd57bcf16f5379ac5e1"
Nov 03 21:09:26.689009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.688961    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq" event={"ID":"9cdca4ca-b9ea-4576-bc2b-21231f397ec9","Type":"ContainerDied","Data":"be03cdb78253a1a8594557d914e2e5fe8412fd2c99ff8b1ad68dee860d8dbd90"}
Nov 03 21:09:26.689525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.689023    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8" event={"ID":"f4d80abf-0dc2-4ba7-9bc6-3692275e9287","Type":"ContainerDied","Data":"836ecf13ae83db7daa2fa006d19d7987c99c9806a2ff1d70d18ef4e26a30d8fb"}
Nov 03 21:09:26.689525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.689083    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"
Nov 03 21:09:26.689525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.689409    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"
Nov 03 21:09:26.718667 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.718624    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9" podUID="6f838b11-fd4b-492c-bf4e-15a5a0bf4208" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:09:26.718667 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.718641    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f" podUID="d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:09:26.905008 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.904962    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"]
Nov 03 21:09:26.909525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:26.909502    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/simulator-w5-1ec892-kserve-7cbc6bd986-jcfj8"]
Nov 03 21:09:27.068699 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:27.068660    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"]
Nov 03 21:09:27.073416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:27.073368    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1ec892/distinct-2-w5-1ec892-kserve-6cccd94bf-8x7sq"]
Nov 03 21:09:28.111459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:28.111426    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" path="/var/lib/kubelet/pods/9cdca4ca-b9ea-4576-bc2b-21231f397ec9/volumes"
Nov 03 21:09:28.111977 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:28.111962    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" path="/var/lib/kubelet/pods/f4d80abf-0dc2-4ba7-9bc6-3692275e9287/volumes"
Nov 03 21:09:28.306704 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:28.306673    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:28.517060 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:28.516944    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:28.912108 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:28.912060    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:29.320688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:29.320644    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:33.200305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:33.200261    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:36.718542 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:36.718500    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9" podUID="6f838b11-fd4b-492c-bf4e-15a5a0bf4208" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:09:36.718982 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:36.718500    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f" podUID="d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:09:37.402306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.402269    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"]
Nov 03 21:09:37.403116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403080    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ef28e282-ce5f-479d-a403-7ea65ef54d24" containerName="revoke-keys"
Nov 03 21:09:37.403180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403117    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef28e282-ce5f-479d-a403-7ea65ef54d24" containerName="revoke-keys"
Nov 03 21:09:37.403180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403134    5196 cpu_manager.go:414] "containerMap: removing container" podUID="4b00d510-0f2b-4896-8cf2-0bcb280bcdb8" containerName="revoke-keys"
Nov 03 21:09:37.403180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403139    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b00d510-0f2b-4896-8cf2-0bcb280bcdb8" containerName="revoke-keys"
Nov 03 21:09:37.403180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403154    5196 cpu_manager.go:414] "containerMap: removing container" podUID="221afa29-4ac5-4bed-8e2b-bc2eb593ac56" containerName="revoke-keys"
Nov 03 21:09:37.403180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403160    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221afa29-4ac5-4bed-8e2b-bc2eb593ac56" containerName="revoke-keys"
Nov 03 21:09:37.403180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403173    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main"
Nov 03 21:09:37.403180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403179    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403200    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3a757404-5bf1-4e94-ba40-2c8ce0d83959" containerName="payload-pre-processing"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403205    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a757404-5bf1-4e94-ba40-2c8ce0d83959" containerName="payload-pre-processing"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403214    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403218    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403229    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ccbfd999-a314-42bd-9c1d-b8bcd1f7e699" containerName="payload-processing"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403233    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbfd999-a314-42bd-9c1d-b8bcd1f7e699" containerName="payload-processing"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403341    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef28e282-ce5f-479d-a403-7ea65ef54d24" containerName="revoke-keys"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403350    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b00d510-0f2b-4896-8cf2-0bcb280bcdb8" containerName="revoke-keys"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403359    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="221afa29-4ac5-4bed-8e2b-bc2eb593ac56" containerName="revoke-keys"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403367    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdca4ca-b9ea-4576-bc2b-21231f397ec9" containerName="main"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403376    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a757404-5bf1-4e94-ba40-2c8ce0d83959" containerName="payload-pre-processing"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403384    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d80abf-0dc2-4ba7-9bc6-3692275e9287" containerName="main"
Nov 03 21:09:37.403407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.403392    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbfd999-a314-42bd-9c1d-b8bcd1f7e699" containerName="payload-processing"
Nov 03 21:09:37.409021 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.409001    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.459068 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.459037    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"]
Nov 03 21:09:37.464649 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.464619    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"]
Nov 03 21:09:37.464831 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.464729    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.500358 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.500327    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:37.500558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.500390    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:09:37.522737 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.522698    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"]
Nov 03 21:09:37.549527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.549489    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjkb8\" (UniqueName: \"kubernetes.io/projected/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-kube-api-access-kjkb8\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.550052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.550021    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.551004 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.550977    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.631182 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.631148    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:37.652298 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.652272    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.652477 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.652366    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjkb8\" (UniqueName: \"kubernetes.io/projected/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-kube-api-access-kjkb8\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.652477 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.652429    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njcct\" (UniqueName: \"kubernetes.io/projected/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-kube-api-access-njcct\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.652477 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.652457    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.652799 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.652778    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.652889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.652873    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.653589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.653563    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.655372 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.655351    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.706354 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.706313    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjkb8\" (UniqueName: \"kubernetes.io/projected/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-kube-api-access-kjkb8\") pod \"maas-api-revoke-sub-c104393c-9rzng\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.726156 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.726080    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:37.754114 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.754064    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.754263 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.754177    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcct\" (UniqueName: \"kubernetes.io/projected/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-kube-api-access-njcct\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.754263 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.754215    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.755029 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.755002    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.756763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.756739    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:37.849927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:37.849814    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcct\" (UniqueName: \"kubernetes.io/projected/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-kube-api-access-njcct\") pod \"maas-api-revoke-sub-66d882e4-mw64t\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:38.080898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:38.080851    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:38.139868 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:38.139834    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"]
Nov 03 21:09:38.140170 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:09:38.139884    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee45821b_f2a2_41e1_8250_bd8c74a7f7e3.slice/crio-e31aad30a737fd76abf69a1f020c173138e42a75d824f25dcfca13120adfae67 WatchSource:0}: Error finding container e31aad30a737fd76abf69a1f020c173138e42a75d824f25dcfca13120adfae67: Status 404 returned error can't find the container with id e31aad30a737fd76abf69a1f020c173138e42a75d824f25dcfca13120adfae67
Nov 03 21:09:38.525684 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:38.525629    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"]
Nov 03 21:09:38.547119 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:09:38.545312    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcc8f5d_09cd_4a04_8213_dee7da6d120c.slice/crio-e65fa8cda01b00a7d924290d8ffc9f58ec91b30d7c456a3741b738fb9d2ce0d9 WatchSource:0}: Error finding container e65fa8cda01b00a7d924290d8ffc9f58ec91b30d7c456a3741b738fb9d2ce0d9: Status 404 returned error can't find the container with id e65fa8cda01b00a7d924290d8ffc9f58ec91b30d7c456a3741b738fb9d2ce0d9
Nov 03 21:09:38.943336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:38.943234    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng" event={"ID":"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3","Type":"ContainerStarted","Data":"90f74c5d1c57653a4650f66294db0cc9c7c92cd6482a2d5f62f71700b71a9d3a"}
Nov 03 21:09:38.943833 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:38.943348    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng" event={"ID":"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3","Type":"ContainerStarted","Data":"e31aad30a737fd76abf69a1f020c173138e42a75d824f25dcfca13120adfae67"}
Nov 03 21:09:38.948915 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:38.948873    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t" event={"ID":"ffcc8f5d-09cd-4a04-8213-dee7da6d120c","Type":"ContainerStarted","Data":"e65fa8cda01b00a7d924290d8ffc9f58ec91b30d7c456a3741b738fb9d2ce0d9"}
Nov 03 21:09:39.058080 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.058018    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng" podStartSLOduration=2.058003501 podStartE2EDuration="2.058003501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:39.054348385 +0000 UTC m=+2419.058307135" watchObservedRunningTime="2026-11-03 21:09:39.058003501 +0000 UTC m=+2419.061962224"
Nov 03 21:09:39.231009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.230908    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:39.541180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.539230    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:39.702259 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.702222    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:39.800576 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.800426    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"]
Nov 03 21:09:39.819857 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.819808    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c571ef-mmff6"]
Nov 03 21:09:39.852545 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.852506    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:39.963617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.963574    5196 generic.go:356] "Generic (PLEG): container finished" podID="ee45821b-f2a2-41e1-8250-bd8c74a7f7e3" containerID="90f74c5d1c57653a4650f66294db0cc9c7c92cd6482a2d5f62f71700b71a9d3a" exitCode=0
Nov 03 21:09:39.964182 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.963657    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng" event={"ID":"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3","Type":"ContainerDied","Data":"90f74c5d1c57653a4650f66294db0cc9c7c92cd6482a2d5f62f71700b71a9d3a"}
Nov 03 21:09:39.966008 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.965984    5196 generic.go:356] "Generic (PLEG): container finished" podID="ffcc8f5d-09cd-4a04-8213-dee7da6d120c" containerID="1a366e42b70e46de91a0c5ddc3220fdf7e65f40bf890e2decad0a381b0181006" exitCode=0
Nov 03 21:09:39.966179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:39.966076    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t" event={"ID":"ffcc8f5d-09cd-4a04-8213-dee7da6d120c","Type":"ContainerDied","Data":"1a366e42b70e46de91a0c5ddc3220fdf7e65f40bf890e2decad0a381b0181006"}
Nov 03 21:09:40.114174 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:40.114086    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afccfc6a-322c-433a-87c0-65e0bf0bf85e" path="/var/lib/kubelet/pods/afccfc6a-322c-433a-87c0-65e0bf0bf85e/volumes"
Nov 03 21:09:40.209161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:40.209125    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:41.167690 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.167649    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:41.214689 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.214643    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:41.247571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.247488    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-xks-ca\") pod \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") "
Nov 03 21:09:41.247721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.247647    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-xks-ca\") pod \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") "
Nov 03 21:09:41.247778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.247720    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-kube-api-access-kjkb8\" (UniqueName: \"kubernetes.io/projected/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-kube-api-access-kjkb8\") pod \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") "
Nov 03 21:09:41.247778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.247759    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-openshift-ca\") pod \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\" (UID: \"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3\") "
Nov 03 21:09:41.247900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.247825    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-kube-api-access-njcct\" (UniqueName: \"kubernetes.io/projected/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-kube-api-access-njcct\") pod \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") "
Nov 03 21:09:41.247900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.247890    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-openshift-ca\") pod \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\" (UID: \"ffcc8f5d-09cd-4a04-8213-dee7da6d120c\") "
Nov 03 21:09:41.248327 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.248298    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-openshift-ca" pod "ee45821b-f2a2-41e1-8250-bd8c74a7f7e3" (UID: "ee45821b-f2a2-41e1-8250-bd8c74a7f7e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:41.248652 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.248618    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-openshift-ca" pod "ffcc8f5d-09cd-4a04-8213-dee7da6d120c" (UID: "ffcc8f5d-09cd-4a04-8213-dee7da6d120c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:41.249024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.248996    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:41.249152 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.249032    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:41.250067 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.250039    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-xks-ca" pod "ffcc8f5d-09cd-4a04-8213-dee7da6d120c" (UID: "ffcc8f5d-09cd-4a04-8213-dee7da6d120c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:41.250198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.250084    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-kube-api-access-kjkb8" pod "ee45821b-f2a2-41e1-8250-bd8c74a7f7e3" (UID: "ee45821b-f2a2-41e1-8250-bd8c74a7f7e3"). InnerVolumeSpecName "kube-api-access-kjkb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:41.250198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.250149    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-xks-ca" pod "ee45821b-f2a2-41e1-8250-bd8c74a7f7e3" (UID: "ee45821b-f2a2-41e1-8250-bd8c74a7f7e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:41.250428 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.250409    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-kube-api-access-njcct" pod "ffcc8f5d-09cd-4a04-8213-dee7da6d120c" (UID: "ffcc8f5d-09cd-4a04-8213-dee7da6d120c"). InnerVolumeSpecName "kube-api-access-njcct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:41.350399 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.350362    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:41.350399 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.350395    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjkb8\" (UniqueName: \"kubernetes.io/projected/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3-kube-api-access-kjkb8\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:41.350399 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.350405    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njcct\" (UniqueName: \"kubernetes.io/projected/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-kube-api-access-njcct\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:41.350642 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.350415    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcc8f5d-09cd-4a04-8213-dee7da6d120c-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:41.897015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.896979    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 21:09:41.961294 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.961261    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-kube-api-access-vvjm5\" (UniqueName: \"kubernetes.io/projected/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-kube-api-access-vvjm5\") pod \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\" (UID: \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\") "
Nov 03 21:09:41.961500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.961378    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-plugins-config-volume\") pod \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\" (UID: \"6f838b11-fd4b-492c-bf4e-15a5a0bf4208\") "
Nov 03 21:09:41.961834 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.961808    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-plugins-config-volume" pod "6f838b11-fd4b-492c-bf4e-15a5a0bf4208" (UID: "6f838b11-fd4b-492c-bf4e-15a5a0bf4208"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:41.962349 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.962319    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:41.964249 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:41.964222    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-kube-api-access-vvjm5" pod "6f838b11-fd4b-492c-bf4e-15a5a0bf4208" (UID: "6f838b11-fd4b-492c-bf4e-15a5a0bf4208"). InnerVolumeSpecName "kube-api-access-vvjm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:42.029302 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.029263    5196 generic.go:356] "Generic (PLEG): container finished" podID="6f838b11-fd4b-492c-bf4e-15a5a0bf4208" containerID="d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8" exitCode=0
Nov 03 21:09:42.029496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.029310    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"
Nov 03 21:09:42.029496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.029418    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9" event={"ID":"6f838b11-fd4b-492c-bf4e-15a5a0bf4208","Type":"ContainerDied","Data":"d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8"}
Nov 03 21:09:42.029496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.029458    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9" event={"ID":"6f838b11-fd4b-492c-bf4e-15a5a0bf4208","Type":"ContainerDied","Data":"65321dec21a55cd7edc93c1f9d9833ac873f3b25dcdacbb2fd51063952552b09"}
Nov 03 21:09:42.029496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.029484    5196 scope.go:122] "RemoveContainer" containerID="d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8"
Nov 03 21:09:42.033597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.033561    5196 generic.go:356] "Generic (PLEG): container finished" podID="d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" containerID="ab8034821ef8c5679dd17813ff9b0f274b69a2421ab6e034d361f02ff2da4932" exitCode=0
Nov 03 21:09:42.033957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.033883    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f" event={"ID":"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859","Type":"ContainerDied","Data":"ab8034821ef8c5679dd17813ff9b0f274b69a2421ab6e034d361f02ff2da4932"}
Nov 03 21:09:42.036120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.036077    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng" event={"ID":"ee45821b-f2a2-41e1-8250-bd8c74a7f7e3","Type":"ContainerDied","Data":"e31aad30a737fd76abf69a1f020c173138e42a75d824f25dcfca13120adfae67"}
Nov 03 21:09:42.036275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.036145    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e31aad30a737fd76abf69a1f020c173138e42a75d824f25dcfca13120adfae67"
Nov 03 21:09:42.036275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.036220    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"
Nov 03 21:09:42.051817 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.050014    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t" event={"ID":"ffcc8f5d-09cd-4a04-8213-dee7da6d120c","Type":"ContainerDied","Data":"e65fa8cda01b00a7d924290d8ffc9f58ec91b30d7c456a3741b738fb9d2ce0d9"}
Nov 03 21:09:42.051817 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.050073    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e65fa8cda01b00a7d924290d8ffc9f58ec91b30d7c456a3741b738fb9d2ce0d9"
Nov 03 21:09:42.051817 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.050169    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"
Nov 03 21:09:42.057876 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.057851    5196 scope.go:122] "RemoveContainer" containerID="d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8"
Nov 03 21:09:42.058720 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:09:42.058693    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8\": container with ID starting with d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8 not found: ID does not exist" containerID="d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8"
Nov 03 21:09:42.058859 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.058729    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8"} err="failed to get container status \"d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8\": rpc error: code = NotFound desc = could not find container \"d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8\": container with ID starting with d645839e7aa0ff579042c3ab5ff4052f3e94f01b5fc0c011ea3e183ff45570a8 not found: ID does not exist"
Nov 03 21:09:42.066005 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.065977    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvjm5\" (UniqueName: \"kubernetes.io/projected/6f838b11-fd4b-492c-bf4e-15a5a0bf4208-kube-api-access-vvjm5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:42.071142 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.071067    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 21:09:42.167085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.166998    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-kube-api-access-gc7tm\" (UniqueName: \"kubernetes.io/projected/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-kube-api-access-gc7tm\") pod \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\" (UID: \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\") "
Nov 03 21:09:42.167290 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.167221    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-plugins-config-volume\") pod \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\" (UID: \"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859\") "
Nov 03 21:09:42.167586 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.167561    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-plugins-config-volume" pod "d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" (UID: "d33d4d1c-a8db-45f3-9796-b6f2c5ee4859"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:42.168016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.167999    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:42.169469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.169447    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-kube-api-access-gc7tm" pod "d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" (UID: "d33d4d1c-a8db-45f3-9796-b6f2c5ee4859"). InnerVolumeSpecName "kube-api-access-gc7tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:42.237450 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.237413    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"]
Nov 03 21:09:42.242490 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.242464    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1ec892-6bd98c48c7-clhc9"]
Nov 03 21:09:42.269320 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:42.269289    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc7tm\" (UniqueName: \"kubernetes.io/projected/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859-kube-api-access-gc7tm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:43.061999 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:43.061963    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"
Nov 03 21:09:43.062231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:43.061962    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f" event={"ID":"d33d4d1c-a8db-45f3-9796-b6f2c5ee4859","Type":"ContainerDied","Data":"7d4cd296f9a43e7035d8195133e67ed0ea355263df883c19ed0beda795476e45"}
Nov 03 21:09:43.062231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:43.062185    5196 scope.go:122] "RemoveContainer" containerID="ab8034821ef8c5679dd17813ff9b0f274b69a2421ab6e034d361f02ff2da4932"
Nov 03 21:09:43.250701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:43.250661    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"]
Nov 03 21:09:43.296635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:43.296586    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1ec892-685d5768c-fns7f"]
Nov 03 21:09:44.110329 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:44.110293    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f838b11-fd4b-492c-bf4e-15a5a0bf4208" path="/var/lib/kubelet/pods/6f838b11-fd4b-492c-bf4e-15a5a0bf4208/volumes"
Nov 03 21:09:44.110745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:44.110730    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" path="/var/lib/kubelet/pods/d33d4d1c-a8db-45f3-9796-b6f2c5ee4859/volumes"
Nov 03 21:09:44.278277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:44.278240    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:44.454597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:44.454486    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:44.789235 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:44.789133    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:45.901180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:45.901141    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:46.060889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:46.060852    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:46.232361 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:46.232275    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:46.859303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:46.859270    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:47.368370 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.368329    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"]
Nov 03 21:09:47.369179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369161    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ee45821b-f2a2-41e1-8250-bd8c74a7f7e3" containerName="revoke-keys"
Nov 03 21:09:47.369179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369179    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee45821b-f2a2-41e1-8250-bd8c74a7f7e3" containerName="revoke-keys"
Nov 03 21:09:47.369303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369234    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ffcc8f5d-09cd-4a04-8213-dee7da6d120c" containerName="revoke-keys"
Nov 03 21:09:47.369303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369240    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcc8f5d-09cd-4a04-8213-dee7da6d120c" containerName="revoke-keys"
Nov 03 21:09:47.369303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369253    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" containerName="payload-pre-processing"
Nov 03 21:09:47.369303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369259    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" containerName="payload-pre-processing"
Nov 03 21:09:47.369303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369267    5196 cpu_manager.go:414] "containerMap: removing container" podUID="6f838b11-fd4b-492c-bf4e-15a5a0bf4208" containerName="payload-processing"
Nov 03 21:09:47.369303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369272    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f838b11-fd4b-492c-bf4e-15a5a0bf4208" containerName="payload-processing"
Nov 03 21:09:47.369496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369381    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcc8f5d-09cd-4a04-8213-dee7da6d120c" containerName="revoke-keys"
Nov 03 21:09:47.369496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369391    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33d4d1c-a8db-45f3-9796-b6f2c5ee4859" containerName="payload-pre-processing"
Nov 03 21:09:47.369496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369397    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f838b11-fd4b-492c-bf4e-15a5a0bf4208" containerName="payload-processing"
Nov 03 21:09:47.369496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.369405    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee45821b-f2a2-41e1-8250-bd8c74a7f7e3" containerName="revoke-keys"
Nov 03 21:09:47.374555 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.374532    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.426797 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.426761    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"]
Nov 03 21:09:47.467080 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.467040    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:47.467080 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.467048    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:09:47.474927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.474894    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.475130 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.475045    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72cz\" (UniqueName: \"kubernetes.io/projected/d09099f2-173b-49d3-b98f-e31aa94749d2-kube-api-access-w72cz\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.475237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.475221    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.577055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.577016    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.577348 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.577204    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72cz\" (UniqueName: \"kubernetes.io/projected/d09099f2-173b-49d3-b98f-e31aa94749d2-kube-api-access-w72cz\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.577424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.577359    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.578045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.577980    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.579741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.579715    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.631039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.630961    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72cz\" (UniqueName: \"kubernetes.io/projected/d09099f2-173b-49d3-b98f-e31aa94749d2-kube-api-access-w72cz\") pod \"maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.690851 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.690822    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:47.866746 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:47.866709    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:48.020572 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:48.020476    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:48.082233 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:48.082196    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"]
Nov 03 21:09:48.087361 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:09:48.087317    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09099f2_173b_49d3_b98f_e31aa94749d2.slice/crio-413634ad0315507d9c378dcac39a46dbe7571118f4b4651519000f1f1b05a7e5 WatchSource:0}: Error finding container 413634ad0315507d9c378dcac39a46dbe7571118f4b4651519000f1f1b05a7e5: Status 404 returned error can't find the container with id 413634ad0315507d9c378dcac39a46dbe7571118f4b4651519000f1f1b05a7e5
Nov 03 21:09:48.142598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:48.142563    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2" event={"ID":"d09099f2-173b-49d3-b98f-e31aa94749d2","Type":"ContainerStarted","Data":"413634ad0315507d9c378dcac39a46dbe7571118f4b4651519000f1f1b05a7e5"}
Nov 03 21:09:48.487404 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:09:48.487361    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:09:48.598750 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:48.598718    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:49.153790 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:49.153751    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2" event={"ID":"d09099f2-173b-49d3-b98f-e31aa94749d2","Type":"ContainerStarted","Data":"73c86e95c42132fe67fff5c889ef38296e0d2d830ce7ee7693a76e57d8792d29"}
Nov 03 21:09:49.191297 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:49.191256    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:49.264884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:49.264825    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2" podStartSLOduration=2.26480757 podStartE2EDuration="2.26480757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:49.261865297 +0000 UTC m=+2429.265824049" watchObservedRunningTime="2026-11-03 21:09:49.26480757 +0000 UTC m=+2429.268766297"
Nov 03 21:09:49.448154 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:49.448033    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:50.182814 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:50.182774    5196 generic.go:356] "Generic (PLEG): container finished" podID="d09099f2-173b-49d3-b98f-e31aa94749d2" containerID="73c86e95c42132fe67fff5c889ef38296e0d2d830ce7ee7693a76e57d8792d29" exitCode=0
Nov 03 21:09:50.183368 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:50.182866    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2" event={"ID":"d09099f2-173b-49d3-b98f-e31aa94749d2","Type":"ContainerDied","Data":"73c86e95c42132fe67fff5c889ef38296e0d2d830ce7ee7693a76e57d8792d29"}
Nov 03 21:09:50.830690 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:50.830648    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:51.078624 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.078591    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:51.221921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.221887    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:51.310017 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.309971    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"]
Nov 03 21:09:51.332020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.331972    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3ca125-6tsq8"]
Nov 03 21:09:51.390873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.390846    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:51.556053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.555971    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-openshift-ca\") pod \"d09099f2-173b-49d3-b98f-e31aa94749d2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") "
Nov 03 21:09:51.556264 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.556080    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-xks-ca\") pod \"d09099f2-173b-49d3-b98f-e31aa94749d2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") "
Nov 03 21:09:51.556264 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.556166    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09099f2-173b-49d3-b98f-e31aa94749d2-kube-api-access-w72cz\" (UniqueName: \"kubernetes.io/projected/d09099f2-173b-49d3-b98f-e31aa94749d2-kube-api-access-w72cz\") pod \"d09099f2-173b-49d3-b98f-e31aa94749d2\" (UID: \"d09099f2-173b-49d3-b98f-e31aa94749d2\") "
Nov 03 21:09:51.556483 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.556449    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-openshift-ca" pod "d09099f2-173b-49d3-b98f-e31aa94749d2" (UID: "d09099f2-173b-49d3-b98f-e31aa94749d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:51.556784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.556769    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:51.558485 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.558455    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-xks-ca" pod "d09099f2-173b-49d3-b98f-e31aa94749d2" (UID: "d09099f2-173b-49d3-b98f-e31aa94749d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:51.558485 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.558457    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09099f2-173b-49d3-b98f-e31aa94749d2-kube-api-access-w72cz" pod "d09099f2-173b-49d3-b98f-e31aa94749d2" (UID: "d09099f2-173b-49d3-b98f-e31aa94749d2"). InnerVolumeSpecName "kube-api-access-w72cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:51.658560 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.658514    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09099f2-173b-49d3-b98f-e31aa94749d2-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:51.658560 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.658552    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72cz\" (UniqueName: \"kubernetes.io/projected/d09099f2-173b-49d3-b98f-e31aa94749d2-kube-api-access-w72cz\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:51.740598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.740558    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:51.744726 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.744696    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"]
Nov 03 21:09:51.922036 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:51.921993    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:52.111996 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:52.111959    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f6b7ac-cd4e-404a-b643-213236e27cde" path="/var/lib/kubelet/pods/82f6b7ac-cd4e-404a-b643-213236e27cde/volumes"
Nov 03 21:09:52.223830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:52.223746    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2" event={"ID":"d09099f2-173b-49d3-b98f-e31aa94749d2","Type":"ContainerDied","Data":"413634ad0315507d9c378dcac39a46dbe7571118f4b4651519000f1f1b05a7e5"}
Nov 03 21:09:52.223830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:52.223798    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="413634ad0315507d9c378dcac39a46dbe7571118f4b4651519000f1f1b05a7e5"
Nov 03 21:09:52.223830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:52.223750    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"
Nov 03 21:09:53.930111 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:53.930065    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:56.183484 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:56.183448    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:57.400773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:57.400720    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:57.603071 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:57.603034    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:57.812377 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:57.812284    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:58.317792 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.317756    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:58.808270 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.808178    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"]
Nov 03 21:09:58.810541 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.810509    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d09099f2-173b-49d3-b98f-e31aa94749d2" containerName="revoke-keys"
Nov 03 21:09:58.810747 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.810731    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09099f2-173b-49d3-b98f-e31aa94749d2" containerName="revoke-keys"
Nov 03 21:09:58.811342 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.811319    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09099f2-173b-49d3-b98f-e31aa94749d2" containerName="revoke-keys"
Nov 03 21:09:58.818748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.818721    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:58.860379 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.860346    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"]
Nov 03 21:09:58.877646 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.877601    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:58.877936 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.877913    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zd75\" (UniqueName: \"kubernetes.io/projected/1b009d39-60db-49fe-8f0e-5953cdf37b17-kube-api-access-5zd75\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:58.878001 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.877972    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:58.901988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.901947    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:58.902193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.902014    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:09:58.979448 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.979407    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:58.979658 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.979637    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zd75\" (UniqueName: \"kubernetes.io/projected/1b009d39-60db-49fe-8f0e-5953cdf37b17-kube-api-access-5zd75\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:58.979714 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.979688    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:58.980321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.980293    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:58.982139 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:58.982082    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:59.033410 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:59.033380    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zd75\" (UniqueName: \"kubernetes.io/projected/1b009d39-60db-49fe-8f0e-5953cdf37b17-kube-api-access-5zd75\") pod \"maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:59.135894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:59.135859    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:09:59.524460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:09:59.524345    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"]
Nov 03 21:09:59.530074 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:09:59.530047    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b009d39_60db_49fe_8f0e_5953cdf37b17.slice/crio-8873f05ff4a785889d4f5ea7deb32255b5ac0575dc94cda4bcf922548e809be1 WatchSource:0}: Error finding container 8873f05ff4a785889d4f5ea7deb32255b5ac0575dc94cda4bcf922548e809be1: Status 404 returned error can't find the container with id 8873f05ff4a785889d4f5ea7deb32255b5ac0575dc94cda4bcf922548e809be1
Nov 03 21:10:00.118875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:00.118842    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"]
Nov 03 21:10:00.231254 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:00.231157    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"]
Nov 03 21:10:00.299653 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:00.299619    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5" event={"ID":"1b009d39-60db-49fe-8f0e-5953cdf37b17","Type":"ContainerStarted","Data":"8bc642887d65b27c49ea06060a0c0e44c99919f73205e6466fab85c4a236c3b8"}
Nov 03 21:10:00.299843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:00.299678    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5" event={"ID":"1b009d39-60db-49fe-8f0e-5953cdf37b17","Type":"ContainerStarted","Data":"8873f05ff4a785889d4f5ea7deb32255b5ac0575dc94cda4bcf922548e809be1"}
Nov 03 21:10:00.450164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:00.450070    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5" podStartSLOduration=2.450049896 podStartE2EDuration="2.450049896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:00.448874354 +0000 UTC m=+2440.452833079" watchObservedRunningTime="2026-11-03 21:10:00.450049896 +0000 UTC m=+2440.454008625"
Nov 03 21:10:00.504014 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:00.503929    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:00.700626 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:00.700564    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:01.310285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:01.310246    5196 generic.go:356] "Generic (PLEG): container finished" podID="1b009d39-60db-49fe-8f0e-5953cdf37b17" containerID="8bc642887d65b27c49ea06060a0c0e44c99919f73205e6466fab85c4a236c3b8" exitCode=0
Nov 03 21:10:01.310753 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:01.310345    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5" event={"ID":"1b009d39-60db-49fe-8f0e-5953cdf37b17","Type":"ContainerDied","Data":"8bc642887d65b27c49ea06060a0c0e44c99919f73205e6466fab85c4a236c3b8"}
Nov 03 21:10:01.361542 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:01.361510    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:02.510053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.510021    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:10:02.551273 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.551236    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-openshift-ca\") pod \"1b009d39-60db-49fe-8f0e-5953cdf37b17\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") "
Nov 03 21:10:02.551465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.551347    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b009d39-60db-49fe-8f0e-5953cdf37b17-kube-api-access-5zd75\" (UniqueName: \"kubernetes.io/projected/1b009d39-60db-49fe-8f0e-5953cdf37b17-kube-api-access-5zd75\") pod \"1b009d39-60db-49fe-8f0e-5953cdf37b17\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") "
Nov 03 21:10:02.551541 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.551532    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-xks-ca\") pod \"1b009d39-60db-49fe-8f0e-5953cdf37b17\" (UID: \"1b009d39-60db-49fe-8f0e-5953cdf37b17\") "
Nov 03 21:10:02.551607 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.551586    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-openshift-ca" pod "1b009d39-60db-49fe-8f0e-5953cdf37b17" (UID: "1b009d39-60db-49fe-8f0e-5953cdf37b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:02.552062 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.552038    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:02.553741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.553710    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-xks-ca" pod "1b009d39-60db-49fe-8f0e-5953cdf37b17" (UID: "1b009d39-60db-49fe-8f0e-5953cdf37b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:02.553843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.553810    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b009d39-60db-49fe-8f0e-5953cdf37b17-kube-api-access-5zd75" pod "1b009d39-60db-49fe-8f0e-5953cdf37b17" (UID: "1b009d39-60db-49fe-8f0e-5953cdf37b17"). InnerVolumeSpecName "kube-api-access-5zd75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:02.653114 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.653013    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zd75\" (UniqueName: \"kubernetes.io/projected/1b009d39-60db-49fe-8f0e-5953cdf37b17-kube-api-access-5zd75\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:02.653114 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:02.653054    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b009d39-60db-49fe-8f0e-5953cdf37b17-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.260277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:03.260243    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"]
Nov 03 21:10:03.308914 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:03.308878    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"]
Nov 03 21:10:03.330012 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:03.329981    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"
Nov 03 21:10:03.330427 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:03.329973    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5" event={"ID":"1b009d39-60db-49fe-8f0e-5953cdf37b17","Type":"ContainerDied","Data":"8873f05ff4a785889d4f5ea7deb32255b5ac0575dc94cda4bcf922548e809be1"}
Nov 03 21:10:03.330427 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:03.330159    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8873f05ff4a785889d4f5ea7deb32255b5ac0575dc94cda4bcf922548e809be1"
Nov 03 21:10:03.434830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:03.434797    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:05.119660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.119612    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n" podUID="e34243db-4751-4a72-ad70-4595e94a4af0" containerName="payload-processing" containerID="cri-o://b9499b5f7f9510c529c2fb6babcd285ce91adb8b5338e24cdc533fa52de5a5dd" gracePeriod=25
Nov 03 21:10:05.232522 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.232475    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z" podUID="ad0688f5-93e5-4dde-b79c-8298e1843c23" containerName="payload-pre-processing" containerID="cri-o://277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22" gracePeriod=25
Nov 03 21:10:05.363594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.363552    5196 generic.go:356] "Generic (PLEG): container finished" podID="e34243db-4751-4a72-ad70-4595e94a4af0" containerID="b9499b5f7f9510c529c2fb6babcd285ce91adb8b5338e24cdc533fa52de5a5dd" exitCode=0
Nov 03 21:10:05.363755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.363631    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n" event={"ID":"e34243db-4751-4a72-ad70-4595e94a4af0","Type":"ContainerDied","Data":"b9499b5f7f9510c529c2fb6babcd285ce91adb8b5338e24cdc533fa52de5a5dd"}
Nov 03 21:10:05.422022 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.421986    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 21:10:05.496898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.496854    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34243db-4751-4a72-ad70-4595e94a4af0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34243db-4751-4a72-ad70-4595e94a4af0-plugins-config-volume\") pod \"e34243db-4751-4a72-ad70-4595e94a4af0\" (UID: \"e34243db-4751-4a72-ad70-4595e94a4af0\") "
Nov 03 21:10:05.497167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.497135    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34243db-4751-4a72-ad70-4595e94a4af0-kube-api-access-d4skh\" (UniqueName: \"kubernetes.io/projected/e34243db-4751-4a72-ad70-4595e94a4af0-kube-api-access-d4skh\") pod \"e34243db-4751-4a72-ad70-4595e94a4af0\" (UID: \"e34243db-4751-4a72-ad70-4595e94a4af0\") "
Nov 03 21:10:05.497351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.497327    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34243db-4751-4a72-ad70-4595e94a4af0-plugins-config-volume" pod "e34243db-4751-4a72-ad70-4595e94a4af0" (UID: "e34243db-4751-4a72-ad70-4595e94a4af0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:05.497801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.497779    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34243db-4751-4a72-ad70-4595e94a4af0-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:05.500487 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.500430    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34243db-4751-4a72-ad70-4595e94a4af0-kube-api-access-d4skh" pod "e34243db-4751-4a72-ad70-4595e94a4af0" (UID: "e34243db-4751-4a72-ad70-4595e94a4af0"). InnerVolumeSpecName "kube-api-access-d4skh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:05.541545 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.541514    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 21:10:05.599295 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.599259    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0688f5-93e5-4dde-b79c-8298e1843c23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0688f5-93e5-4dde-b79c-8298e1843c23-plugins-config-volume\") pod \"ad0688f5-93e5-4dde-b79c-8298e1843c23\" (UID: \"ad0688f5-93e5-4dde-b79c-8298e1843c23\") "
Nov 03 21:10:05.599505 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.599356    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0688f5-93e5-4dde-b79c-8298e1843c23-kube-api-access-t6dng\" (UniqueName: \"kubernetes.io/projected/ad0688f5-93e5-4dde-b79c-8298e1843c23-kube-api-access-t6dng\") pod \"ad0688f5-93e5-4dde-b79c-8298e1843c23\" (UID: \"ad0688f5-93e5-4dde-b79c-8298e1843c23\") "
Nov 03 21:10:05.599708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.599669    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0688f5-93e5-4dde-b79c-8298e1843c23-plugins-config-volume" pod "ad0688f5-93e5-4dde-b79c-8298e1843c23" (UID: "ad0688f5-93e5-4dde-b79c-8298e1843c23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:05.600108 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.600034    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0688f5-93e5-4dde-b79c-8298e1843c23-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:05.600108 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.600059    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4skh\" (UniqueName: \"kubernetes.io/projected/e34243db-4751-4a72-ad70-4595e94a4af0-kube-api-access-d4skh\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:05.601741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.601716    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0688f5-93e5-4dde-b79c-8298e1843c23-kube-api-access-t6dng" pod "ad0688f5-93e5-4dde-b79c-8298e1843c23" (UID: "ad0688f5-93e5-4dde-b79c-8298e1843c23"). InnerVolumeSpecName "kube-api-access-t6dng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:05.701792 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.701691    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6dng\" (UniqueName: \"kubernetes.io/projected/ad0688f5-93e5-4dde-b79c-8298e1843c23-kube-api-access-t6dng\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:05.712257 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.712222    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"]
Nov 03 21:10:05.712991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.712976    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ad0688f5-93e5-4dde-b79c-8298e1843c23" containerName="payload-pre-processing"
Nov 03 21:10:05.713035 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.712993    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0688f5-93e5-4dde-b79c-8298e1843c23" containerName="payload-pre-processing"
Nov 03 21:10:05.713035 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.713011    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e34243db-4751-4a72-ad70-4595e94a4af0" containerName="payload-processing"
Nov 03 21:10:05.713035 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.713016    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34243db-4751-4a72-ad70-4595e94a4af0" containerName="payload-processing"
Nov 03 21:10:05.713168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.713038    5196 cpu_manager.go:414] "containerMap: removing container" podUID="1b009d39-60db-49fe-8f0e-5953cdf37b17" containerName="revoke-keys"
Nov 03 21:10:05.713168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.713044    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b009d39-60db-49fe-8f0e-5953cdf37b17" containerName="revoke-keys"
Nov 03 21:10:05.713247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.713204    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b009d39-60db-49fe-8f0e-5953cdf37b17" containerName="revoke-keys"
Nov 03 21:10:05.713247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.713228    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0688f5-93e5-4dde-b79c-8298e1843c23" containerName="payload-pre-processing"
Nov 03 21:10:05.713247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.713246    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34243db-4751-4a72-ad70-4595e94a4af0" containerName="payload-processing"
Nov 03 21:10:05.716943 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.716918    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.767988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.767947    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"]
Nov 03 21:10:05.771823 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.771789    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:10:05.772020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.771897    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:10:05.774865 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.774837    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:05.776374 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.776342    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"]
Nov 03 21:10:05.805727 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.805674    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.805992 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.805972    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.806051 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.806034    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t624\" (UniqueName: \"kubernetes.io/projected/2aee95c0-42b1-4005-acec-4ef171f602cc-kube-api-access-9t624\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.836164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.831115    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"]
Nov 03 21:10:05.907753 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.907721    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlj2\" (UniqueName: \"kubernetes.io/projected/33c2834e-5f21-4460-a456-856a6df586b3-kube-api-access-qjlj2\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:05.907939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.907787    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.907984 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.907940    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:05.908146 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.908086    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.908217 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.908171    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t624\" (UniqueName: \"kubernetes.io/projected/2aee95c0-42b1-4005-acec-4ef171f602cc-kube-api-access-9t624\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.908217 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.908192    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:05.908783 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.908758    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.910471 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.910446    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:05.944143 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.944085    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:05.962144 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:05.962049    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t624\" (UniqueName: \"kubernetes.io/projected/2aee95c0-42b1-4005-acec-4ef171f602cc-kube-api-access-9t624\") pod \"maas-api-revoke-sub-29c463e0-jcwt2\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:06.009936 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.009899    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:06.010143 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.010035    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:06.010143 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.010088    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlj2\" (UniqueName: \"kubernetes.io/projected/33c2834e-5f21-4460-a456-856a6df586b3-kube-api-access-qjlj2\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:06.010633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.010612    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:06.012863 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.012841    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:06.035370 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.035328    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:06.069197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.069046    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlj2\" (UniqueName: \"kubernetes.io/projected/33c2834e-5f21-4460-a456-856a6df586b3-kube-api-access-qjlj2\") pod \"maas-api-revoke-sub-e0c1753c-6dgcf\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:06.106267 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.105966    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:06.164310 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.160392    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:06.383544 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.383501    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n" event={"ID":"e34243db-4751-4a72-ad70-4595e94a4af0","Type":"ContainerDied","Data":"b4a253e57db05c55cac36f05825a9094ff3d8f132497c3f63241137fc709b514"}
Nov 03 21:10:06.383836 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.383816    5196 scope.go:122] "RemoveContainer" containerID="b9499b5f7f9510c529c2fb6babcd285ce91adb8b5338e24cdc533fa52de5a5dd"
Nov 03 21:10:06.384196 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.384161    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"
Nov 03 21:10:06.393355 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.393315    5196 generic.go:356] "Generic (PLEG): container finished" podID="ad0688f5-93e5-4dde-b79c-8298e1843c23" containerID="277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22" exitCode=0
Nov 03 21:10:06.393509 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.393425    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z" event={"ID":"ad0688f5-93e5-4dde-b79c-8298e1843c23","Type":"ContainerDied","Data":"277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22"}
Nov 03 21:10:06.393583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.393558    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z" event={"ID":"ad0688f5-93e5-4dde-b79c-8298e1843c23","Type":"ContainerDied","Data":"e4ce615001110edd9c5543d6e5749de1c1ccd9dbd6857688fdb8fa971062942a"}
Nov 03 21:10:06.393697 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.393678    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"
Nov 03 21:10:06.407434 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.407414    5196 scope.go:122] "RemoveContainer" containerID="277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22"
Nov 03 21:10:06.423501 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.423458    5196 scope.go:122] "RemoveContainer" containerID="277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22"
Nov 03 21:10:06.423965 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:06.423930    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22\": container with ID starting with 277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22 not found: ID does not exist" containerID="277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22"
Nov 03 21:10:06.424380 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.423973    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22"} err="failed to get container status \"277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22\": rpc error: code = NotFound desc = could not find container \"277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22\": container with ID starting with 277f07ac8e4f8b35ca7cd15cf58309f7ca950032e2f0cce1e957e15239212b22 not found: ID does not exist"
Nov 03 21:10:06.490674 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:10:06.490641    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aee95c0_42b1_4005_acec_4ef171f602cc.slice/crio-43cc32e4c0e1e98a3c023bf940f5a34e6f6ee94289dcf18bd240acbff6508e11 WatchSource:0}: Error finding container 43cc32e4c0e1e98a3c023bf940f5a34e6f6ee94289dcf18bd240acbff6508e11: Status 404 returned error can't find the container with id 43cc32e4c0e1e98a3c023bf940f5a34e6f6ee94289dcf18bd240acbff6508e11
Nov 03 21:10:06.491965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.491939    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"]
Nov 03 21:10:06.546504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.546460    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"]
Nov 03 21:10:06.552963 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.552932    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fd9afe-7bbbd4777-sw57n"]
Nov 03 21:10:06.633063 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.632881    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:06.634057 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.634027    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"]
Nov 03 21:10:06.656369 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.656324    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"]
Nov 03 21:10:06.661956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.661928    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fd9afe-7c56667b79-6729z"]
Nov 03 21:10:06.767871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:06.767832    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main" containerID="cri-o://de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688" gracePeriod=45
Nov 03 21:10:07.077597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.077569    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 21:10:07.184141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.183992    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5af553-68f7-4fd1-830b-c334ec31edc0-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5af553-68f7-4fd1-830b-c334ec31edc0-tls-certs\") pod \"5e5af553-68f7-4fd1-830b-c334ec31edc0\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") "
Nov 03 21:10:07.184141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.184054    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5af553-68f7-4fd1-830b-c334ec31edc0-kube-api-access-6r4bz\" (UniqueName: \"kubernetes.io/projected/5e5af553-68f7-4fd1-830b-c334ec31edc0-kube-api-access-6r4bz\") pod \"5e5af553-68f7-4fd1-830b-c334ec31edc0\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") "
Nov 03 21:10:07.184141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.184124    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-home\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-home\") pod \"5e5af553-68f7-4fd1-830b-c334ec31edc0\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") "
Nov 03 21:10:07.184755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.184254    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-tmp-dir\") pod \"5e5af553-68f7-4fd1-830b-c334ec31edc0\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") "
Nov 03 21:10:07.184755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.184413    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-model-cache\") pod \"5e5af553-68f7-4fd1-830b-c334ec31edc0\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") "
Nov 03 21:10:07.186980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.184966    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-dshm\") pod \"5e5af553-68f7-4fd1-830b-c334ec31edc0\" (UID: \"5e5af553-68f7-4fd1-830b-c334ec31edc0\") "
Nov 03 21:10:07.186980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.185632    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-model-cache" pod "5e5af553-68f7-4fd1-830b-c334ec31edc0" (UID: "5e5af553-68f7-4fd1-830b-c334ec31edc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:07.186980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.185884    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:07.186980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.185957    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-tmp-dir" pod "5e5af553-68f7-4fd1-830b-c334ec31edc0" (UID: "5e5af553-68f7-4fd1-830b-c334ec31edc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:07.187418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.187384    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-home" pod "5e5af553-68f7-4fd1-830b-c334ec31edc0" (UID: "5e5af553-68f7-4fd1-830b-c334ec31edc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:07.191420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.191067    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5af553-68f7-4fd1-830b-c334ec31edc0-kube-api-access-6r4bz" pod "5e5af553-68f7-4fd1-830b-c334ec31edc0" (UID: "5e5af553-68f7-4fd1-830b-c334ec31edc0"). InnerVolumeSpecName "kube-api-access-6r4bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:07.191420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.191338    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5af553-68f7-4fd1-830b-c334ec31edc0-tls-certs" pod "5e5af553-68f7-4fd1-830b-c334ec31edc0" (UID: "5e5af553-68f7-4fd1-830b-c334ec31edc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:07.191696 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.191491    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-dshm" pod "5e5af553-68f7-4fd1-830b-c334ec31edc0" (UID: "5e5af553-68f7-4fd1-830b-c334ec31edc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:07.287887 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.287850    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:07.287887 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.287890    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:07.288140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.287905    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e5af553-68f7-4fd1-830b-c334ec31edc0-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:07.288140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.287920    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r4bz\" (UniqueName: \"kubernetes.io/projected/5e5af553-68f7-4fd1-830b-c334ec31edc0-kube-api-access-6r4bz\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:07.288140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.287942    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e5af553-68f7-4fd1-830b-c334ec31edc0-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:07.405290 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.405174    5196 generic.go:356] "Generic (PLEG): container finished" podID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerID="de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688" exitCode=0
Nov 03 21:10:07.405290 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.405224    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"
Nov 03 21:10:07.405290 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.405260    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" event={"ID":"5e5af553-68f7-4fd1-830b-c334ec31edc0","Type":"ContainerDied","Data":"de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688"}
Nov 03 21:10:07.405623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.405309    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w" event={"ID":"5e5af553-68f7-4fd1-830b-c334ec31edc0","Type":"ContainerDied","Data":"77b64375713e21c7cb6eae2a43113cee64b5c291617d34ff576d8a8319d15efa"}
Nov 03 21:10:07.405623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.405334    5196 scope.go:122] "RemoveContainer" containerID="de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688"
Nov 03 21:10:07.407365 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.407336    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf" event={"ID":"33c2834e-5f21-4460-a456-856a6df586b3","Type":"ContainerStarted","Data":"b2f2c292071aebfdcd8c53ab1ffb59be6e7c0ce6862e5e76179bed50378f7f02"}
Nov 03 21:10:07.407505 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.407390    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf" event={"ID":"33c2834e-5f21-4460-a456-856a6df586b3","Type":"ContainerStarted","Data":"c15df525db3cbf99910e13714765543609b212581aa2ce565e7937511b2f6dfb"}
Nov 03 21:10:07.409855 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.409830    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2" event={"ID":"2aee95c0-42b1-4005-acec-4ef171f602cc","Type":"ContainerStarted","Data":"7277dda51a43854e7191d876fe0365d675bc231d123c24bb56c26ae91054e0d0"}
Nov 03 21:10:07.409993 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.409883    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2" event={"ID":"2aee95c0-42b1-4005-acec-4ef171f602cc","Type":"ContainerStarted","Data":"43cc32e4c0e1e98a3c023bf940f5a34e6f6ee94289dcf18bd240acbff6508e11"}
Nov 03 21:10:07.418377 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.418358    5196 scope.go:122] "RemoveContainer" containerID="de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688"
Nov 03 21:10:07.418640 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:07.418622    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688\": container with ID starting with de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688 not found: ID does not exist" containerID="de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688"
Nov 03 21:10:07.418691 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.418648    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688"} err="failed to get container status \"de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688\": rpc error: code = NotFound desc = could not find container \"de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688\": container with ID starting with de4908775e866eaae2bcc8c866e421b1263546c501c9d0a27189fe0b51b7a688 not found: ID does not exist"
Nov 03 21:10:07.537285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.537182    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:07.562548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.562505    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"]
Nov 03 21:10:07.566862 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.566841    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fd9afe/simulator-w0-fd9afe-kserve-5f95f6d4d8-4pv2w"]
Nov 03 21:10:07.669649 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.669600    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2" podStartSLOduration=2.669584929 podStartE2EDuration="2.669584929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:07.668484797 +0000 UTC m=+2447.672443521" watchObservedRunningTime="2026-11-03 21:10:07.669584929 +0000 UTC m=+2447.673543705"
Nov 03 21:10:07.778853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.778801    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf" podStartSLOduration=2.778786243 podStartE2EDuration="2.778786243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:07.777585892 +0000 UTC m=+2447.781544643" watchObservedRunningTime="2026-11-03 21:10:07.778786243 +0000 UTC m=+2447.782745042"
Nov 03 21:10:07.787696 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.787609    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:07.942437 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:07.942399    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:08.012672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.012628    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"]
Nov 03 21:10:08.017382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.017348    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a19980-9dhp2"]
Nov 03 21:10:08.111731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.111695    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c2044d-b1a0-48fd-8299-e47a349d16b2" path="/var/lib/kubelet/pods/01c2044d-b1a0-48fd-8299-e47a349d16b2/volumes"
Nov 03 21:10:08.112237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.112221    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" path="/var/lib/kubelet/pods/5e5af553-68f7-4fd1-830b-c334ec31edc0/volumes"
Nov 03 21:10:08.112637 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.112623    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0688f5-93e5-4dde-b79c-8298e1843c23" path="/var/lib/kubelet/pods/ad0688f5-93e5-4dde-b79c-8298e1843c23/volumes"
Nov 03 21:10:08.113418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.113400    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34243db-4751-4a72-ad70-4595e94a4af0" path="/var/lib/kubelet/pods/e34243db-4751-4a72-ad70-4595e94a4af0/volumes"
Nov 03 21:10:08.423600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.423507    5196 generic.go:356] "Generic (PLEG): container finished" podID="33c2834e-5f21-4460-a456-856a6df586b3" containerID="b2f2c292071aebfdcd8c53ab1ffb59be6e7c0ce6862e5e76179bed50378f7f02" exitCode=0
Nov 03 21:10:08.424028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.423595    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf" event={"ID":"33c2834e-5f21-4460-a456-856a6df586b3","Type":"ContainerDied","Data":"b2f2c292071aebfdcd8c53ab1ffb59be6e7c0ce6862e5e76179bed50378f7f02"}
Nov 03 21:10:08.425296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.425259    5196 generic.go:356] "Generic (PLEG): container finished" podID="2aee95c0-42b1-4005-acec-4ef171f602cc" containerID="7277dda51a43854e7191d876fe0365d675bc231d123c24bb56c26ae91054e0d0" exitCode=0
Nov 03 21:10:08.425418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.425325    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2" event={"ID":"2aee95c0-42b1-4005-acec-4ef171f602cc","Type":"ContainerDied","Data":"7277dda51a43854e7191d876fe0365d675bc231d123c24bb56c26ae91054e0d0"}
Nov 03 21:10:08.537617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:08.537575    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:09.619583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.619470    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:09.658664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.658631    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:09.723167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.723055    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-xks-ca\") pod \"2aee95c0-42b1-4005-acec-4ef171f602cc\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") "
Nov 03 21:10:09.723355 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.723177    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-openshift-ca\") pod \"33c2834e-5f21-4460-a456-856a6df586b3\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") "
Nov 03 21:10:09.723355 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.723343    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-openshift-ca\") pod \"2aee95c0-42b1-4005-acec-4ef171f602cc\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") "
Nov 03 21:10:09.723481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.723391    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aee95c0-42b1-4005-acec-4ef171f602cc-kube-api-access-9t624\" (UniqueName: \"kubernetes.io/projected/2aee95c0-42b1-4005-acec-4ef171f602cc-kube-api-access-9t624\") pod \"2aee95c0-42b1-4005-acec-4ef171f602cc\" (UID: \"2aee95c0-42b1-4005-acec-4ef171f602cc\") "
Nov 03 21:10:09.723481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.723463    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c2834e-5f21-4460-a456-856a6df586b3-kube-api-access-qjlj2\" (UniqueName: \"kubernetes.io/projected/33c2834e-5f21-4460-a456-856a6df586b3-kube-api-access-qjlj2\") pod \"33c2834e-5f21-4460-a456-856a6df586b3\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") "
Nov 03 21:10:09.723592 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.723491    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-xks-ca\") pod \"33c2834e-5f21-4460-a456-856a6df586b3\" (UID: \"33c2834e-5f21-4460-a456-856a6df586b3\") "
Nov 03 21:10:09.723669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.723638    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-openshift-ca" pod "33c2834e-5f21-4460-a456-856a6df586b3" (UID: "33c2834e-5f21-4460-a456-856a6df586b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:09.723732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.723696    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-openshift-ca" pod "2aee95c0-42b1-4005-acec-4ef171f602cc" (UID: "2aee95c0-42b1-4005-acec-4ef171f602cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:09.724498 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.724460    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:09.724628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.724504    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:09.726282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.726247    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aee95c0-42b1-4005-acec-4ef171f602cc-kube-api-access-9t624" pod "2aee95c0-42b1-4005-acec-4ef171f602cc" (UID: "2aee95c0-42b1-4005-acec-4ef171f602cc"). InnerVolumeSpecName "kube-api-access-9t624". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:09.726282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.726264    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-xks-ca" pod "33c2834e-5f21-4460-a456-856a6df586b3" (UID: "33c2834e-5f21-4460-a456-856a6df586b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:09.726447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.726413    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-xks-ca" pod "2aee95c0-42b1-4005-acec-4ef171f602cc" (UID: "2aee95c0-42b1-4005-acec-4ef171f602cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:09.726611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.726525    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c2834e-5f21-4460-a456-856a6df586b3-kube-api-access-qjlj2" pod "33c2834e-5f21-4460-a456-856a6df586b3" (UID: "33c2834e-5f21-4460-a456-856a6df586b3"). InnerVolumeSpecName "kube-api-access-qjlj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:09.826402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.826367    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjlj2\" (UniqueName: \"kubernetes.io/projected/33c2834e-5f21-4460-a456-856a6df586b3-kube-api-access-qjlj2\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:09.826402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.826400    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c2834e-5f21-4460-a456-856a6df586b3-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:09.826628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.826410    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aee95c0-42b1-4005-acec-4ef171f602cc-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:09.826628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:09.826420    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t624\" (UniqueName: \"kubernetes.io/projected/2aee95c0-42b1-4005-acec-4ef171f602cc-kube-api-access-9t624\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:10.405065 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.404965    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"]
Nov 03 21:10:10.444476 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.444442    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf" event={"ID":"33c2834e-5f21-4460-a456-856a6df586b3","Type":"ContainerDied","Data":"c15df525db3cbf99910e13714765543609b212581aa2ce565e7937511b2f6dfb"}
Nov 03 21:10:10.444651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.444497    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15df525db3cbf99910e13714765543609b212581aa2ce565e7937511b2f6dfb"
Nov 03 21:10:10.444651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.444451    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:10:10.446234 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.446210    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"
Nov 03 21:10:10.446357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.446240    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2" event={"ID":"2aee95c0-42b1-4005-acec-4ef171f602cc","Type":"ContainerDied","Data":"43cc32e4c0e1e98a3c023bf940f5a34e6f6ee94289dcf18bd240acbff6508e11"}
Nov 03 21:10:10.446357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.446284    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43cc32e4c0e1e98a3c023bf940f5a34e6f6ee94289dcf18bd240acbff6508e11"
Nov 03 21:10:10.506892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.506791    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"]
Nov 03 21:10:10.781229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.781145    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:10.936968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:10.936864    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:12.321187 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.321153    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:12.526012 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.525933    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:12.629059 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.628966    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"]
Nov 03 21:10:12.629267 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.629216    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy" containerID="cri-o://7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e" gracePeriod=30
Nov 03 21:10:12.646184 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.646141    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"]
Nov 03 21:10:12.646386 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.646364    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy" containerID="cri-o://816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9" gracePeriod=30
Nov 03 21:10:12.670675 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.670639    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"]
Nov 03 21:10:12.670929 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.670900    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy" containerID="cri-o://1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61" gracePeriod=30
Nov 03 21:10:12.682055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:12.682013    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:13.088529 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:13.088497    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:14.891616 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:14.891576    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:15.043565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:15.043512    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:15.216018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:15.215924    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"]
Nov 03 21:10:15.216223 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:15.216175    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy" containerID="cri-o://fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243" gracePeriod=30
Nov 03 21:10:15.245795 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:15.245758    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:15.405533 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:15.405490    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw" podUID="b092dd7c-4650-406a-b0b5-216381fd4a07" containerName="payload-processing" containerID="cri-o://4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a" gracePeriod=25
Nov 03 21:10:15.508044 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:15.507946    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt" podUID="51e873ca-6f50-4ad7-b052-74ffdbcd72cc" containerName="payload-pre-processing" containerID="cri-o://0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89" gracePeriod=25
Nov 03 21:10:17.013884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:17.013851    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:17.191499 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:17.191456    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:17.598176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:17.598138    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:10:17.864694 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:17.864541    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:18.104903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.103702    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:10:18.198851 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.198823    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:10:18.271806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.265849    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-data\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.279141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.272112    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94587bc3-1388-4ef1-aab9-a16c547680df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94587bc3-1388-4ef1-aab9-a16c547680df-istiod-ca-cert\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.279441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279415    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-credential-socket\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.279597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279458    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-certs\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.279597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279493    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-certs\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.279597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279551    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-kube-api-access-tnsxt\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-kube-api-access-tnsxt\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.279771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279606    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-token\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-token\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.279771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279642    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94587bc3-1388-4ef1-aab9-a16c547680df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94587bc3-1388-4ef1-aab9-a16c547680df-istio-podinfo\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.279771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279691    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-envoy\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.279771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279747    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-credential-socket\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.279974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279794    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-podinfo\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.279974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279825    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-socket\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.279974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279939    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-kube-api-access-vkxfm\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-kube-api-access-vkxfm\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.280168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.279993    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-istio-token\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-istio-token\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.280168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.280066    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-envoy\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.280281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.280174    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-socket\") pod \"94587bc3-1388-4ef1-aab9-a16c547680df\" (UID: \"94587bc3-1388-4ef1-aab9-a16c547680df\") "
Nov 03 21:10:18.280281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.280234    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad18a218-f0e9-4880-b7b9-cce51ac03077-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18a218-f0e9-4880-b7b9-cce51ac03077-istiod-ca-cert\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.280381 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.266206    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-data" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.280381 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.272597    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94587bc3-1388-4ef1-aab9-a16c547680df-istiod-ca-cert" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:18.280528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.280502    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-credential-socket" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.280803 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.280780    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-certs" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.281045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.281025    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-certs" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.282013 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.281241    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-socket" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.282013 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.281692    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad18a218-f0e9-4880-b7b9-cce51ac03077-istiod-ca-cert" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:18.282341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282320    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.282449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282348    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18a218-f0e9-4880-b7b9-cce51ac03077-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.282449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282361    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.282449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282376    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94587bc3-1388-4ef1-aab9-a16c547680df-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.282449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282390    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.282449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282406    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.282449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282421    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.283037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282647    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-socket" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.283037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.282699    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-credential-socket" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.285823 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.285195    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main" containerID="cri-o://e5be51f5a587da6dbb3f8e9956458c96017512a2569d66a4b74c45950f34e915" gracePeriod=45
Nov 03 21:10:18.286814 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.286790    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-envoy" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.287889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.287825    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-kube-api-access-tnsxt" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "kube-api-access-tnsxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:18.287889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.287863    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-kube-api-access-vkxfm" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "kube-api-access-vkxfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:18.288059 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.287904    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-token" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:18.288335 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.288300    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-podinfo" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:18.292453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.291812    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-istio-token" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:18.292453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.292205    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-envoy" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.293078 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.293041    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94587bc3-1388-4ef1-aab9-a16c547680df-istio-podinfo" pod "94587bc3-1388-4ef1-aab9-a16c547680df" (UID: "94587bc3-1388-4ef1-aab9-a16c547680df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:18.300418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.300255    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:10:18.332581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.332529    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main" containerID="cri-o://ebd740fce9cb9c0e5edd029257e144634d15d2b5dfde55bdd7b43b37ff855826" gracePeriod=45
Nov 03 21:10:18.389197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.385640    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istiod-ca-cert\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.389197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.385725    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-data\") pod \"ad18a218-f0e9-4880-b7b9-cce51ac03077\" (UID: \"ad18a218-f0e9-4880-b7b9-cce51ac03077\") "
Nov 03 21:10:18.389197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.385761    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-token\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-token\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.389197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.385801    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-credential-socket\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.389197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.385862    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-data\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.389197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.386008    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-envoy\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.392679 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.389896    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-socket\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.392679 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.390012    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-podinfo\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.392679 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.390073    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-certs\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.392679 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.390153    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-kube-api-access-m4wjc\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-kube-api-access-m4wjc\") pod \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\" (UID: \"2613d8e3-3f14-4278-9b00-30dbfd9246d8\") "
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403081    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnsxt\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-kube-api-access-tnsxt\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403139    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403158    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94587bc3-1388-4ef1-aab9-a16c547680df-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403172    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403187    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403200    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403216    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403230    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxfm\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-kube-api-access-vkxfm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403243    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94587bc3-1388-4ef1-aab9-a16c547680df-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.403257    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94587bc3-1388-4ef1-aab9-a16c547680df-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.409490    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-credential-socket" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.411423    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istiod-ca-cert" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.414146    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-data" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.416553    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-socket" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.423907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.416701    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-certs" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.460762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.447192    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-data" pod "ad18a218-f0e9-4880-b7b9-cce51ac03077" (UID: "ad18a218-f0e9-4880-b7b9-cce51ac03077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.475322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.475256    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-envoy" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:18.475975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.475931    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-podinfo" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:18.479785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.479738    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-token" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:18.480078 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.480047    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-kube-api-access-m4wjc" pod "2613d8e3-3f14-4278-9b00-30dbfd9246d8" (UID: "2613d8e3-3f14-4278-9b00-30dbfd9246d8"). InnerVolumeSpecName "kube-api-access-m4wjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509411    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509451    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509466    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509482    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wjc\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-kube-api-access-m4wjc\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509497    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509511    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18a218-f0e9-4880-b7b9-cce51ac03077-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509524    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509537    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509550    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.509635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.509565    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2613d8e3-3f14-4278-9b00-30dbfd9246d8-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:18.564716 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:18.564669    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:10:18.573891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.573849    5196 generic.go:356] "Generic (PLEG): container finished" podID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerID="816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9" exitCode=0
Nov 03 21:10:18.574067 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.573962    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" event={"ID":"ad18a218-f0e9-4880-b7b9-cce51ac03077","Type":"ContainerDied","Data":"816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9"}
Nov 03 21:10:18.574067 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.574020    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp" event={"ID":"ad18a218-f0e9-4880-b7b9-cce51ac03077","Type":"ContainerDied","Data":"cad469ad9514a0251fe25e746e7387b55be745339b8caba7967df21b9059c84b"}
Nov 03 21:10:18.574067 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.574056    5196 scope.go:122] "RemoveContainer" containerID="816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9"
Nov 03 21:10:18.574285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.574266    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"
Nov 03 21:10:18.692683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.652405    5196 generic.go:356] "Generic (PLEG): container finished" podID="94587bc3-1388-4ef1-aab9-a16c547680df" containerID="7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e" exitCode=0
Nov 03 21:10:18.692683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.652601    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" event={"ID":"94587bc3-1388-4ef1-aab9-a16c547680df","Type":"ContainerDied","Data":"7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e"}
Nov 03 21:10:18.692683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.652654    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr" event={"ID":"94587bc3-1388-4ef1-aab9-a16c547680df","Type":"ContainerDied","Data":"dca0078a445118b85bef125c8e864a1554b873d89e7a4271ba474b833b0dc0c3"}
Nov 03 21:10:18.692683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.652755    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"
Nov 03 21:10:18.706244 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.704934    5196 generic.go:356] "Generic (PLEG): container finished" podID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerID="1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61" exitCode=0
Nov 03 21:10:18.706244 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.705047    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" event={"ID":"2613d8e3-3f14-4278-9b00-30dbfd9246d8","Type":"ContainerDied","Data":"1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61"}
Nov 03 21:10:18.706244 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.705121    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq" event={"ID":"2613d8e3-3f14-4278-9b00-30dbfd9246d8","Type":"ContainerDied","Data":"668fcf5cdbab30a7175d64e17732e368987d47662e2ed6551fcd919c4b5cd778"}
Nov 03 21:10:18.706244 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.705222    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"
Nov 03 21:10:18.711002 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.710971    5196 scope.go:122] "RemoveContainer" containerID="816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9"
Nov 03 21:10:18.711628 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:18.711593    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9\": container with ID starting with 816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9 not found: ID does not exist" containerID="816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9"
Nov 03 21:10:18.711724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.711643    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9"} err="failed to get container status \"816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9\": rpc error: code = NotFound desc = could not find container \"816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9\": container with ID starting with 816ba79032a29433f8e705fc47d76b60904d7cb87972ca5570706b5fc9559aa9 not found: ID does not exist"
Nov 03 21:10:18.711724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.711670    5196 scope.go:122] "RemoveContainer" containerID="7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e"
Nov 03 21:10:18.712360 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.712335    5196 generic.go:356] "Generic (PLEG): container finished" podID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerID="ebd740fce9cb9c0e5edd029257e144634d15d2b5dfde55bdd7b43b37ff855826" exitCode=0
Nov 03 21:10:18.712563 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.712538    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" event={"ID":"163cd80e-8575-45d0-89ff-6163f25c57e7","Type":"ContainerDied","Data":"ebd740fce9cb9c0e5edd029257e144634d15d2b5dfde55bdd7b43b37ff855826"}
Nov 03 21:10:18.720328 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.720178    5196 generic.go:356] "Generic (PLEG): container finished" podID="3eee8222-1165-47be-ab0f-237afafa9d07" containerID="e5be51f5a587da6dbb3f8e9956458c96017512a2569d66a4b74c45950f34e915" exitCode=0
Nov 03 21:10:18.720328 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.720241    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" event={"ID":"3eee8222-1165-47be-ab0f-237afafa9d07","Type":"ContainerDied","Data":"e5be51f5a587da6dbb3f8e9956458c96017512a2569d66a4b74c45950f34e915"}
Nov 03 21:10:18.750375 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.750022    5196 scope.go:122] "RemoveContainer" containerID="7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e"
Nov 03 21:10:18.750523 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:18.750494    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e\": container with ID starting with 7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e not found: ID does not exist" containerID="7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e"
Nov 03 21:10:18.750583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.750536    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e"} err="failed to get container status \"7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e\": rpc error: code = NotFound desc = could not find container \"7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e\": container with ID starting with 7ee823163b6a1dd0f7a6a4120697e3c57cc515e0209a9d184b315b4d344aa30e not found: ID does not exist"
Nov 03 21:10:18.750583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.750562    5196 scope.go:122] "RemoveContainer" containerID="1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61"
Nov 03 21:10:18.770115 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.769119    5196 scope.go:122] "RemoveContainer" containerID="1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61"
Nov 03 21:10:18.770115 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:18.769525    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61\": container with ID starting with 1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61 not found: ID does not exist" containerID="1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61"
Nov 03 21:10:18.770115 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.769559    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61"} err="failed to get container status \"1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61\": rpc error: code = NotFound desc = could not find container \"1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61\": container with ID starting with 1bf8479cc98d48752a805fbe0723f1bc629059c35763039d4a854ad665e1ff61 not found: ID does not exist"
Nov 03 21:10:18.853284 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.853232    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"]
Nov 03 21:10:18.865405 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.865369    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-8zgjr"]
Nov 03 21:10:18.941881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.941789    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 21:10:18.962846 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.962551    5196 patch_prober.go:29] interesting pod/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:18.962846 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:18.962607    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:19.018179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.018138    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 21:10:19.020854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.020802    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"]
Nov 03 21:10:19.030440 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.030392    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-v5lgq"]
Nov 03 21:10:19.056593 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.056557    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/163cd80e-8575-45d0-89ff-6163f25c57e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/163cd80e-8575-45d0-89ff-6163f25c57e7-tls-certs\") pod \"163cd80e-8575-45d0-89ff-6163f25c57e7\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") "
Nov 03 21:10:19.056830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.056618    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-tmp-dir\") pod \"163cd80e-8575-45d0-89ff-6163f25c57e7\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") "
Nov 03 21:10:19.056830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.056639    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-dshm\") pod \"163cd80e-8575-45d0-89ff-6163f25c57e7\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") "
Nov 03 21:10:19.056830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.056690    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-home\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-home\") pod \"163cd80e-8575-45d0-89ff-6163f25c57e7\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") "
Nov 03 21:10:19.057024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.056946    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-model-cache\") pod \"163cd80e-8575-45d0-89ff-6163f25c57e7\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") "
Nov 03 21:10:19.057024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.056983    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163cd80e-8575-45d0-89ff-6163f25c57e7-kube-api-access-kt6rk\" (UniqueName: \"kubernetes.io/projected/163cd80e-8575-45d0-89ff-6163f25c57e7-kube-api-access-kt6rk\") pod \"163cd80e-8575-45d0-89ff-6163f25c57e7\" (UID: \"163cd80e-8575-45d0-89ff-6163f25c57e7\") "
Nov 03 21:10:19.057024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.056981    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-tmp-dir" pod "163cd80e-8575-45d0-89ff-6163f25c57e7" (UID: "163cd80e-8575-45d0-89ff-6163f25c57e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:19.057236 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.057025    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-home" pod "163cd80e-8575-45d0-89ff-6163f25c57e7" (UID: "163cd80e-8575-45d0-89ff-6163f25c57e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:19.057236 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.057201    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-model-cache" pod "163cd80e-8575-45d0-89ff-6163f25c57e7" (UID: "163cd80e-8575-45d0-89ff-6163f25c57e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:19.057748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.057649    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.057748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.057681    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.057748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.057697    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.059180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.059153    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-dshm" pod "163cd80e-8575-45d0-89ff-6163f25c57e7" (UID: "163cd80e-8575-45d0-89ff-6163f25c57e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:19.059289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.059211    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163cd80e-8575-45d0-89ff-6163f25c57e7-tls-certs" pod "163cd80e-8575-45d0-89ff-6163f25c57e7" (UID: "163cd80e-8575-45d0-89ff-6163f25c57e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:19.059337 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.059318    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163cd80e-8575-45d0-89ff-6163f25c57e7-kube-api-access-kt6rk" pod "163cd80e-8575-45d0-89ff-6163f25c57e7" (UID: "163cd80e-8575-45d0-89ff-6163f25c57e7"). InnerVolumeSpecName "kube-api-access-kt6rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:19.131732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.131694    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"]
Nov 03 21:10:19.138594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.138557    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-211d68bf-openshift-default-588997c885-lncvp"]
Nov 03 21:10:19.158723 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.158630    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eee8222-1165-47be-ab0f-237afafa9d07-kube-api-access-mzbrr\" (UniqueName: \"kubernetes.io/projected/3eee8222-1165-47be-ab0f-237afafa9d07-kube-api-access-mzbrr\") pod \"3eee8222-1165-47be-ab0f-237afafa9d07\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") "
Nov 03 21:10:19.158926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.158807    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-model-cache\") pod \"3eee8222-1165-47be-ab0f-237afafa9d07\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") "
Nov 03 21:10:19.158926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.158899    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eee8222-1165-47be-ab0f-237afafa9d07-tls-certs\" (UniqueName: \"kubernetes.io/secret/3eee8222-1165-47be-ab0f-237afafa9d07-tls-certs\") pod \"3eee8222-1165-47be-ab0f-237afafa9d07\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") "
Nov 03 21:10:19.159075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.158953    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-dshm\") pod \"3eee8222-1165-47be-ab0f-237afafa9d07\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") "
Nov 03 21:10:19.159075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.158992    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-home\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-home\") pod \"3eee8222-1165-47be-ab0f-237afafa9d07\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") "
Nov 03 21:10:19.159217 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.159120    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-model-cache" pod "3eee8222-1165-47be-ab0f-237afafa9d07" (UID: "3eee8222-1165-47be-ab0f-237afafa9d07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:19.159217 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.159148    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-tmp-dir\") pod \"3eee8222-1165-47be-ab0f-237afafa9d07\" (UID: \"3eee8222-1165-47be-ab0f-237afafa9d07\") "
Nov 03 21:10:19.159325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.159297    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-home" pod "3eee8222-1165-47be-ab0f-237afafa9d07" (UID: "3eee8222-1165-47be-ab0f-237afafa9d07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:19.159449 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.159419    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-tmp-dir" pod "3eee8222-1165-47be-ab0f-237afafa9d07" (UID: "3eee8222-1165-47be-ab0f-237afafa9d07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:19.159981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.159955    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163cd80e-8575-45d0-89ff-6163f25c57e7-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.160120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.159986    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163cd80e-8575-45d0-89ff-6163f25c57e7-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.160120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.160001    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.160120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.160015    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.160120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.160030    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.160120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.160046    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt6rk\" (UniqueName: \"kubernetes.io/projected/163cd80e-8575-45d0-89ff-6163f25c57e7-kube-api-access-kt6rk\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.161346 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.161308    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eee8222-1165-47be-ab0f-237afafa9d07-kube-api-access-mzbrr" pod "3eee8222-1165-47be-ab0f-237afafa9d07" (UID: "3eee8222-1165-47be-ab0f-237afafa9d07"). InnerVolumeSpecName "kube-api-access-mzbrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:19.161720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.161701    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-dshm" pod "3eee8222-1165-47be-ab0f-237afafa9d07" (UID: "3eee8222-1165-47be-ab0f-237afafa9d07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:19.161803 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.161723    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eee8222-1165-47be-ab0f-237afafa9d07-tls-certs" pod "3eee8222-1165-47be-ab0f-237afafa9d07" (UID: "3eee8222-1165-47be-ab0f-237afafa9d07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:19.262140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.262086    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eee8222-1165-47be-ab0f-237afafa9d07-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.262140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.262141    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzbrr\" (UniqueName: \"kubernetes.io/projected/3eee8222-1165-47be-ab0f-237afafa9d07-kube-api-access-mzbrr\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.262416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.262158    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eee8222-1165-47be-ab0f-237afafa9d07-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.732859 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.732823    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p" event={"ID":"163cd80e-8575-45d0-89ff-6163f25c57e7","Type":"ContainerDied","Data":"85d1d7f73cbd1ad5a12da48df5496ecea11da2003166d9cffdf0d7a916a2acbb"}
Nov 03 21:10:19.733039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.732882    5196 scope.go:122] "RemoveContainer" containerID="ebd740fce9cb9c0e5edd029257e144634d15d2b5dfde55bdd7b43b37ff855826"
Nov 03 21:10:19.733039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.732838    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"
Nov 03 21:10:19.734599 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.734580    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"
Nov 03 21:10:19.734710 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.734670    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n" event={"ID":"3eee8222-1165-47be-ab0f-237afafa9d07","Type":"ContainerDied","Data":"a9bb3c1e5bc8a522ea8a36762532e9bfa542be4d8947fbd9208d2a791e1f5075"}
Nov 03 21:10:19.746374 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.746353    5196 scope.go:122] "RemoveContainer" containerID="e5be51f5a587da6dbb3f8e9956458c96017512a2569d66a4b74c45950f34e915"
Nov 03 21:10:19.891060 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.891016    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"]
Nov 03 21:10:19.938181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.938064    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/distinct-2-w2-b119f2-kserve-f9f788895-jzd2n"]
Nov 03 21:10:19.997751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:19.997711    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"]
Nov 03 21:10:20.002696 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.002655    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b119f2/simulator-w2-b119f2-kserve-5d954846d-8jv8p"]
Nov 03 21:10:20.111823 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.111787    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" path="/var/lib/kubelet/pods/163cd80e-8575-45d0-89ff-6163f25c57e7/volumes"
Nov 03 21:10:20.112370 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.112350    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" path="/var/lib/kubelet/pods/2613d8e3-3f14-4278-9b00-30dbfd9246d8/volumes"
Nov 03 21:10:20.112766 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.112752    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" path="/var/lib/kubelet/pods/3eee8222-1165-47be-ab0f-237afafa9d07/volumes"
Nov 03 21:10:20.113537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.113520    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" path="/var/lib/kubelet/pods/94587bc3-1388-4ef1-aab9-a16c547680df/volumes"
Nov 03 21:10:20.113956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.113942    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" path="/var/lib/kubelet/pods/ad18a218-f0e9-4880-b7b9-cce51ac03077/volumes"
Nov 03 21:10:20.503515 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.503451    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 21:10:20.582241 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582204    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c8e3ff-db29-4702-b133-7442d29d1ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c8e3ff-db29-4702-b133-7442d29d1ae8-istiod-ca-cert\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.582417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582292    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-podinfo\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.582417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582340    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-token\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.582566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582462    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-certs\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.582566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582481    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-credential-socket\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.582566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582501    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-envoy\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.582566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582546    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-socket\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.582785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582626    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-kube-api-access-vkfmf\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-kube-api-access-vkfmf\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.582785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582649    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c8e3ff-db29-4702-b133-7442d29d1ae8-istiod-ca-cert" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:20.582910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582741    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-certs" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:20.582910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582777    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-credential-socket" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:20.582910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582719    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-data\") pod \"89c8e3ff-db29-4702-b133-7442d29d1ae8\" (UID: \"89c8e3ff-db29-4702-b133-7442d29d1ae8\") "
Nov 03 21:10:20.583054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.582914    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-data" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:20.583054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.583038    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-socket" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:20.583755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.583725    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.583879 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.583758    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.583879 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.583776    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c8e3ff-db29-4702-b133-7442d29d1ae8-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.583879 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.583790    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.583879 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.583809    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.584886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.584839    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-token" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:20.584994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.584940    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-envoy" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:20.585077 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.585055    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-podinfo" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:20.585237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.585221    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-kube-api-access-vkfmf" pod "89c8e3ff-db29-4702-b133-7442d29d1ae8" (UID: "89c8e3ff-db29-4702-b133-7442d29d1ae8"). InnerVolumeSpecName "kube-api-access-vkfmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:20.684932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.684895    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.684932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.684930    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.684932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.684939    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c8e3ff-db29-4702-b133-7442d29d1ae8-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.685209 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.684949    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkfmf\" (UniqueName: \"kubernetes.io/projected/89c8e3ff-db29-4702-b133-7442d29d1ae8-kube-api-access-vkfmf\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.748373 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.748335    5196 generic.go:356] "Generic (PLEG): container finished" podID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerID="fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243" exitCode=0
Nov 03 21:10:20.748543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.748386    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"
Nov 03 21:10:20.748543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.748421    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" event={"ID":"89c8e3ff-db29-4702-b133-7442d29d1ae8","Type":"ContainerDied","Data":"fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243"}
Nov 03 21:10:20.748543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.748476    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj" event={"ID":"89c8e3ff-db29-4702-b133-7442d29d1ae8","Type":"ContainerDied","Data":"38426a367093d0985454e01b37f58e4ac74ba72decf45ea7341696288efe737c"}
Nov 03 21:10:20.748543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.748497    5196 scope.go:122] "RemoveContainer" containerID="fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243"
Nov 03 21:10:20.761513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.761495    5196 scope.go:122] "RemoveContainer" containerID="fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243"
Nov 03 21:10:20.761798 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:20.761776    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243\": container with ID starting with fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243 not found: ID does not exist" containerID="fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243"
Nov 03 21:10:20.761862 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.761811    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243"} err="failed to get container status \"fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243\": rpc error: code = NotFound desc = could not find container \"fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243\": container with ID starting with fa9900fac4808e85db836812a7fafb00793319c04c7714888f009c5c737da243 not found: ID does not exist"
Nov 03 21:10:20.905467 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.905425    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"]
Nov 03 21:10:20.951998 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:20.951964    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a19980-openshift-default-696866dd7c-gdknj"]
Nov 03 21:10:21.759500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:21.759463    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw" podUID="b092dd7c-4650-406a-b0b5-216381fd4a07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:21.759944 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:21.759506    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt" podUID="51e873ca-6f50-4ad7-b052-74ffdbcd72cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:22.111513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:22.111476    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" path="/var/lib/kubelet/pods/89c8e3ff-db29-4702-b133-7442d29d1ae8/volumes"
Nov 03 21:10:22.664700 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:22.664652    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:26.348120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:26.348059    5196 scope.go:122] "RemoveContainer" containerID="4d8f2536f8bedefe8bb97e24e038683557228cb4e03a8a448f9eea6604bad082"
Nov 03 21:10:26.359489 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:26.359468    5196 scope.go:122] "RemoveContainer" containerID="3c5ad09d9ad339e8d295152a46e1000215742531bf2ef1a17852d557e6ea91e5"
Nov 03 21:10:26.379878 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:26.379853    5196 scope.go:122] "RemoveContainer" containerID="b5600df1c8e9f98458a1958e90cd5a192da16c79527c72741d0d970a392ec48d"
Nov 03 21:10:26.390941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:26.390905    5196 scope.go:122] "RemoveContainer" containerID="ac33964eb02eb1917e86279c3a8c25b00084fa08b840c0b668046c1a093dc6ce"
Nov 03 21:10:26.401568 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:26.401544    5196 scope.go:122] "RemoveContainer" containerID="00fbe0e3e528257ac8124a379906ca2b2eea70c044dd4f94562b4888f63e43b0"
Nov 03 21:10:27.445903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:27.445863    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:28.329128 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:28.329071    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:28.530773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:28.530740    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:29.205448 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:29.205410    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:31.760078 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:31.760037    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw" podUID="b092dd7c-4650-406a-b0b5-216381fd4a07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:31.760650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:31.760037    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt" podUID="51e873ca-6f50-4ad7-b052-74ffdbcd72cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:32.330915 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:32.330868    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:34.768191 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:34.768142    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:35.703615 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.703537    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 21:10:35.794932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.794899    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b092dd7c-4650-406a-b0b5-216381fd4a07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092dd7c-4650-406a-b0b5-216381fd4a07-plugins-config-volume\") pod \"b092dd7c-4650-406a-b0b5-216381fd4a07\" (UID: \"b092dd7c-4650-406a-b0b5-216381fd4a07\") "
Nov 03 21:10:35.795480 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.795054    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b092dd7c-4650-406a-b0b5-216381fd4a07-kube-api-access-4rzxn\" (UniqueName: \"kubernetes.io/projected/b092dd7c-4650-406a-b0b5-216381fd4a07-kube-api-access-4rzxn\") pod \"b092dd7c-4650-406a-b0b5-216381fd4a07\" (UID: \"b092dd7c-4650-406a-b0b5-216381fd4a07\") "
Nov 03 21:10:35.795480 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.795411    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b092dd7c-4650-406a-b0b5-216381fd4a07-plugins-config-volume" pod "b092dd7c-4650-406a-b0b5-216381fd4a07" (UID: "b092dd7c-4650-406a-b0b5-216381fd4a07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:35.795623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.795574    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092dd7c-4650-406a-b0b5-216381fd4a07-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:35.798211 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.798176    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b092dd7c-4650-406a-b0b5-216381fd4a07-kube-api-access-4rzxn" pod "b092dd7c-4650-406a-b0b5-216381fd4a07" (UID: "b092dd7c-4650-406a-b0b5-216381fd4a07"). InnerVolumeSpecName "kube-api-access-4rzxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:35.798488 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.798473    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 21:10:35.892910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.892878    5196 generic.go:356] "Generic (PLEG): container finished" podID="b092dd7c-4650-406a-b0b5-216381fd4a07" containerID="4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a" exitCode=0
Nov 03 21:10:35.893104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.892951    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"
Nov 03 21:10:35.893104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.892958    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw" event={"ID":"b092dd7c-4650-406a-b0b5-216381fd4a07","Type":"ContainerDied","Data":"4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a"}
Nov 03 21:10:35.893104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.893078    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw" event={"ID":"b092dd7c-4650-406a-b0b5-216381fd4a07","Type":"ContainerDied","Data":"b2bc738000e96617281cfeb1b116c5fb1fdaa19ee796ab6278b1de8ee14dbd3c"}
Nov 03 21:10:35.893274 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.893121    5196 scope.go:122] "RemoveContainer" containerID="4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a"
Nov 03 21:10:35.894966 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.894940    5196 generic.go:356] "Generic (PLEG): container finished" podID="51e873ca-6f50-4ad7-b052-74ffdbcd72cc" containerID="0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89" exitCode=0
Nov 03 21:10:35.895145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.894982    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"
Nov 03 21:10:35.895145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.894982    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt" event={"ID":"51e873ca-6f50-4ad7-b052-74ffdbcd72cc","Type":"ContainerDied","Data":"0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89"}
Nov 03 21:10:35.895145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.895034    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt" event={"ID":"51e873ca-6f50-4ad7-b052-74ffdbcd72cc","Type":"ContainerDied","Data":"d16392ed3a06ed971f4c3b69d6411c5c60fb60570a631e21aa6f001759f9c20a"}
Nov 03 21:10:35.896413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.896390    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-kube-api-access-8wns8\" (UniqueName: \"kubernetes.io/projected/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-kube-api-access-8wns8\") pod \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\" (UID: \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\") "
Nov 03 21:10:35.896537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.896518    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-plugins-config-volume\") pod \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\" (UID: \"51e873ca-6f50-4ad7-b052-74ffdbcd72cc\") "
Nov 03 21:10:35.896891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.896865    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-plugins-config-volume" pod "51e873ca-6f50-4ad7-b052-74ffdbcd72cc" (UID: "51e873ca-6f50-4ad7-b052-74ffdbcd72cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:35.897155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.897135    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:35.897222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.897162    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rzxn\" (UniqueName: \"kubernetes.io/projected/b092dd7c-4650-406a-b0b5-216381fd4a07-kube-api-access-4rzxn\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:35.898828 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.898807    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-kube-api-access-8wns8" pod "51e873ca-6f50-4ad7-b052-74ffdbcd72cc" (UID: "51e873ca-6f50-4ad7-b052-74ffdbcd72cc"). InnerVolumeSpecName "kube-api-access-8wns8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:35.906531 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.906511    5196 scope.go:122] "RemoveContainer" containerID="4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a"
Nov 03 21:10:35.906804 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:35.906787    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a\": container with ID starting with 4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a not found: ID does not exist" containerID="4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a"
Nov 03 21:10:35.906852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.906813    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a"} err="failed to get container status \"4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a\": rpc error: code = NotFound desc = could not find container \"4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a\": container with ID starting with 4b81dc3577eb586cda8bcc63af641f33d06109bb85c5c9ea24354b75acf0482a not found: ID does not exist"
Nov 03 21:10:35.906852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.906832    5196 scope.go:122] "RemoveContainer" containerID="0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89"
Nov 03 21:10:35.918109 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.918063    5196 scope.go:122] "RemoveContainer" containerID="0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89"
Nov 03 21:10:35.918569 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:35.918537    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89\": container with ID starting with 0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89 not found: ID does not exist" containerID="0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89"
Nov 03 21:10:35.918704 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.918577    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89"} err="failed to get container status \"0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89\": rpc error: code = NotFound desc = could not find container \"0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89\": container with ID starting with 0ee1730d04696ab150bb982e44019dc6071df20e7f21861facabb71de78e5d89 not found: ID does not exist"
Nov 03 21:10:35.998988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:35.998937    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wns8\" (UniqueName: \"kubernetes.io/projected/51e873ca-6f50-4ad7-b052-74ffdbcd72cc-kube-api-access-8wns8\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:36.078400 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:36.078361    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"]
Nov 03 21:10:36.084082 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:36.084036    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b119f2-859d6cf757-nb5kw"]
Nov 03 21:10:36.085245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:36.085219    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:36.114406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:36.114374    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b092dd7c-4650-406a-b0b5-216381fd4a07" path="/var/lib/kubelet/pods/b092dd7c-4650-406a-b0b5-216381fd4a07/volumes"
Nov 03 21:10:36.309238 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:36.309150    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"]
Nov 03 21:10:36.313341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:36.313306    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b119f2-84f8ffbff-sksgt"]
Nov 03 21:10:36.943490 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:36.943447    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:38.112580 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:38.112545    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e873ca-6f50-4ad7-b052-74ffdbcd72cc" path="/var/lib/kubelet/pods/51e873ca-6f50-4ad7-b052-74ffdbcd72cc/volumes"
Nov 03 21:10:38.562166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:38.562037    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:38.783763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:38.783717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:38.951742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:38.951703    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:39.087057 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:39.087024    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"]
Nov 03 21:10:39.134037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:39.134002    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1ec892-c6d5z"]
Nov 03 21:10:39.496055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:39.496021    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:40.112614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:40.112579    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221afa29-4ac5-4bed-8e2b-bc2eb593ac56" path="/var/lib/kubelet/pods/221afa29-4ac5-4bed-8e2b-bc2eb593ac56/volumes"
Nov 03 21:10:40.119377 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:40.119347    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:40.280926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:40.280888    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:41.199175 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:41.197376    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:44.063033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:44.062996    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:45.220883 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:45.220850    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:45.379024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:45.378991    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:45.641162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:45.641123    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"]
Nov 03 21:10:45.641427 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:45.641403    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" containerID="cri-o://1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61" gracePeriod=30
Nov 03 21:10:45.648338 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:45.648306    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"]
Nov 03 21:10:45.648579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:45.648554    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" containerID="cri-o://f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442" gracePeriod=30
Nov 03 21:10:45.659894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:45.659859    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:46.940988 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:46.940955    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:47.044365 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:47.044328    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"]
Nov 03 21:10:47.091195 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:47.091156    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5e3e9c-lklbf"]
Nov 03 21:10:47.895752 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:47.895717    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:47.896002 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:47.895787    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:48.110620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:48.110588    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2824fef-5afd-40af-ac3d-ec66583fcac1" path="/var/lib/kubelet/pods/f2824fef-5afd-40af-ac3d-ec66583fcac1/volumes"
Nov 03 21:10:48.486644 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:48.486610    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:10:49.116456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:49.116409    5196 patch_prober.go:29] interesting pod/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:49.116861 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:49.116493    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:49.353240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:49.353202    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:49.584725 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:49.584685    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:49.783428 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:49.783383    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:50.302968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:50.302935    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"]
Nov 03 21:10:50.303435 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:50.303149    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" containerID="cri-o://f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8" gracePeriod=30
Nov 03 21:10:50.951525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:50.951495    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 21:10:50.983392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:50.983362    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 21:10:51.026814 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.026778    5196 generic.go:356] "Generic (PLEG): container finished" podID="e1a8017f-edde-4061-836f-43dc01941446" containerID="1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61" exitCode=0
Nov 03 21:10:51.027010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.026815    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" event={"ID":"e1a8017f-edde-4061-836f-43dc01941446","Type":"ContainerDied","Data":"1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61"}
Nov 03 21:10:51.027010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.026831    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"
Nov 03 21:10:51.027010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.026884    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b" event={"ID":"e1a8017f-edde-4061-836f-43dc01941446","Type":"ContainerDied","Data":"1d4ff53037a42c0e5027e33cef7c9565f33f7604b3b851f85838ae8755ddfbf3"}
Nov 03 21:10:51.027010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.026919    5196 scope.go:122] "RemoveContainer" containerID="1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61"
Nov 03 21:10:51.028680 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.028658    5196 generic.go:356] "Generic (PLEG): container finished" podID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerID="f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442" exitCode=0
Nov 03 21:10:51.028795 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.028705    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" event={"ID":"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6","Type":"ContainerDied","Data":"f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442"}
Nov 03 21:10:51.028795 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.028739    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl" event={"ID":"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6","Type":"ContainerDied","Data":"74751759c38f25bb17a32ec984186f2561fbc7df2930af207cdd7f5fb92746b3"}
Nov 03 21:10:51.028795 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.028708    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"
Nov 03 21:10:51.030985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.030964    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istiod-ca-cert\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.031057 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031015    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-data\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.031127 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031064    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-kube-api-access-dxs2h\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-kube-api-access-dxs2h\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.031127 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031085    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-envoy\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.031314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031290    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-certs\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.031418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031370    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-credential-socket\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.031418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031379    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-data" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.031418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031411    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istiod-ca-cert" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:51.031656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031497    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-data\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.031656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031533    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a8017f-edde-4061-836f-43dc01941446-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a8017f-edde-4061-836f-43dc01941446-istiod-ca-cert\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.031656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031568    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-socket\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.031656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031612    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-podinfo\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.031656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031652    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-credential-socket" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.032116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031666    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-istio-token\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-istio-token\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.032116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031702    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-socket\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.032116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031753    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-envoy\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.032116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031802    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-token\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-token\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.032116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031831    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-credential-socket\") pod \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\" (UID: \"c8c3a3a5-1657-4d9f-875e-ac5e00c808a6\") "
Nov 03 21:10:51.032116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031840    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-socket" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.032116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031872    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-certs\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.032116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031910    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-kube-api-access-zpbq5\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-kube-api-access-zpbq5\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.032911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031855    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-certs" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.032911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031743    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-data" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.032911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.032024    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-socket" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.032911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.032049    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a8017f-edde-4061-836f-43dc01941446-istiod-ca-cert" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:51.032911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.031978    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1a8017f-edde-4061-836f-43dc01941446-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a8017f-edde-4061-836f-43dc01941446-istio-podinfo\") pod \"e1a8017f-edde-4061-836f-43dc01941446\" (UID: \"e1a8017f-edde-4061-836f-43dc01941446\") "
Nov 03 21:10:51.032911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.032317    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-certs" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.032994    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.033019    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.033038    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.033052    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.032221    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-credential-socket" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.033067    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a8017f-edde-4061-836f-43dc01941446-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.033177    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.033193    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.033208    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.033292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.033236    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.034169 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.034072    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-envoy" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.034336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.034306    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-kube-api-access-dxs2h" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "kube-api-access-dxs2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:51.034893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.034866    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1a8017f-edde-4061-836f-43dc01941446-istio-podinfo" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:51.035040 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.035021    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-podinfo" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:51.035558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.035528    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-istio-token" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:51.035638 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.035555    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-token" pod "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" (UID: "c8c3a3a5-1657-4d9f-875e-ac5e00c808a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:51.036258 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.036239    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-envoy" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:51.037220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.037202    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-kube-api-access-zpbq5" pod "e1a8017f-edde-4061-836f-43dc01941446" (UID: "e1a8017f-edde-4061-836f-43dc01941446"). InnerVolumeSpecName "kube-api-access-zpbq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:51.050336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.050199    5196 scope.go:122] "RemoveContainer" containerID="1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61"
Nov 03 21:10:51.050830 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:51.050800    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61\": container with ID starting with 1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61 not found: ID does not exist" containerID="1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61"
Nov 03 21:10:51.050972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.050841    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61"} err="failed to get container status \"1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61\": rpc error: code = NotFound desc = could not find container \"1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61\": container with ID starting with 1446528fd7f0119c7e67b03f961ccd702cff4e0b54eb8e7d410ce0ebb3ec5c61 not found: ID does not exist"
Nov 03 21:10:51.050972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.050866    5196 scope.go:122] "RemoveContainer" containerID="f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442"
Nov 03 21:10:51.063989 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.063961    5196 scope.go:122] "RemoveContainer" containerID="f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442"
Nov 03 21:10:51.064398 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:51.064368    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442\": container with ID starting with f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442 not found: ID does not exist" containerID="f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442"
Nov 03 21:10:51.064522 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.064409    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442"} err="failed to get container status \"f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442\": rpc error: code = NotFound desc = could not find container \"f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442\": container with ID starting with f62a017215d5a0e862b9b8aefb37f43f5a62d0602825fff1075e84f43794c442 not found: ID does not exist"
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134466    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134501    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134518    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a8017f-edde-4061-836f-43dc01941446-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134532    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134547    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134561    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpbq5\" (UniqueName: \"kubernetes.io/projected/e1a8017f-edde-4061-836f-43dc01941446-kube-api-access-zpbq5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134576    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a8017f-edde-4061-836f-43dc01941446-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134590    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxs2h\" (UniqueName: \"kubernetes.io/projected/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-kube-api-access-dxs2h\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.134502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.134607    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:51.445157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.445122    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"]
Nov 03 21:10:51.449335 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.449306    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-twt7b"]
Nov 03 21:10:51.551731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.551693    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"]
Nov 03 21:10:51.557397 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:51.557364    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1ec892-openshift-default-66cb96fcf-j5ptl"]
Nov 03 21:10:52.110711 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:52.110681    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" path="/var/lib/kubelet/pods/c8c3a3a5-1657-4d9f-875e-ac5e00c808a6/volumes"
Nov 03 21:10:52.111227 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:52.111212    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a8017f-edde-4061-836f-43dc01941446" path="/var/lib/kubelet/pods/e1a8017f-edde-4061-836f-43dc01941446/volumes"
Nov 03 21:10:52.944513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:52.944476    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:52.945005 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:52.944537    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:53.624304 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.624266    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"]
Nov 03 21:10:53.625016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.624997    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy"
Nov 03 21:10:53.625016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625016    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625029    5196 cpu_manager.go:414] "containerMap: removing container" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625034    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625052    5196 cpu_manager.go:414] "containerMap: removing container" podUID="33c2834e-5f21-4460-a456-856a6df586b3" containerName="revoke-keys"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625058    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c2834e-5f21-4460-a456-856a6df586b3" containerName="revoke-keys"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625066    5196 cpu_manager.go:414] "containerMap: removing container" podUID="51e873ca-6f50-4ad7-b052-74ffdbcd72cc" containerName="payload-pre-processing"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625071    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e873ca-6f50-4ad7-b052-74ffdbcd72cc" containerName="payload-pre-processing"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625105    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625110    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625117    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625121    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625129    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b092dd7c-4650-406a-b0b5-216381fd4a07" containerName="payload-processing"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625133    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b092dd7c-4650-406a-b0b5-216381fd4a07" containerName="payload-processing"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625141    5196 cpu_manager.go:414] "containerMap: removing container" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625145    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625152    5196 cpu_manager.go:414] "containerMap: removing container" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625156    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625161    5196 cpu_manager.go:414] "containerMap: removing container" podUID="2aee95c0-42b1-4005-acec-4ef171f602cc" containerName="revoke-keys"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625165    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aee95c0-42b1-4005-acec-4ef171f602cc" containerName="revoke-keys"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625180    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625184    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy"
Nov 03 21:10:53.625181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625190    5196 cpu_manager.go:414] "containerMap: removing container" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625195    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625203    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625208    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625313    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c3a3a5-1657-4d9f-875e-ac5e00c808a6" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625325    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a8017f-edde-4061-836f-43dc01941446" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625330    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="2613d8e3-3f14-4278-9b00-30dbfd9246d8" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625339    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c2834e-5f21-4460-a456-856a6df586b3" containerName="revoke-keys"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625346    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e873ca-6f50-4ad7-b052-74ffdbcd72cc" containerName="payload-pre-processing"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625359    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5af553-68f7-4fd1-830b-c334ec31edc0" containerName="main"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625365    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eee8222-1165-47be-ab0f-237afafa9d07" containerName="main"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625370    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b092dd7c-4650-406a-b0b5-216381fd4a07" containerName="payload-processing"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625375    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="163cd80e-8575-45d0-89ff-6163f25c57e7" containerName="main"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625380    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="94587bc3-1388-4ef1-aab9-a16c547680df" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625386    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aee95c0-42b1-4005-acec-4ef171f602cc" containerName="revoke-keys"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625392    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad18a218-f0e9-4880-b7b9-cce51ac03077" containerName="istio-proxy"
Nov 03 21:10:53.625875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.625396    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c8e3ff-db29-4702-b133-7442d29d1ae8" containerName="istio-proxy"
Nov 03 21:10:53.630812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.630793    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.668593 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.668552    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91547460-ae34-4873-81d3-c18bc132a837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.668789 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.668704    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtz6f\" (UniqueName: \"kubernetes.io/projected/91547460-ae34-4873-81d3-c18bc132a837-kube-api-access-jtz6f\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.668844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.668817    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91547460-ae34-4873-81d3-c18bc132a837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.677496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.677438    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"]
Nov 03 21:10:53.682942 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.682917    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"]
Nov 03 21:10:53.683116 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.683044    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.722478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.722443    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:10:53.722478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.722462    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:10:53.735638 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.735600    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"]
Nov 03 21:10:53.770351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.770315    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51807c06-0f48-443f-ae03-43040b440a23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.770589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.770368    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51807c06-0f48-443f-ae03-43040b440a23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.770589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.770499    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91547460-ae34-4873-81d3-c18bc132a837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.770589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.770537    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbnm\" (UniqueName: \"kubernetes.io/projected/51807c06-0f48-443f-ae03-43040b440a23-kube-api-access-dfbnm\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.770787 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.770676    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91547460-ae34-4873-81d3-c18bc132a837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.770843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.770784    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz6f\" (UniqueName: \"kubernetes.io/projected/91547460-ae34-4873-81d3-c18bc132a837-kube-api-access-jtz6f\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.771794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.771764    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91547460-ae34-4873-81d3-c18bc132a837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.776551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.776517    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91547460-ae34-4873-81d3-c18bc132a837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.824926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.824895    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz6f\" (UniqueName: \"kubernetes.io/projected/91547460-ae34-4873-81d3-c18bc132a837-kube-api-access-jtz6f\") pod \"maas-api-revoke-sub-40a87926-b4j2f\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:53.872620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.872578    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51807c06-0f48-443f-ae03-43040b440a23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.872620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.872619    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51807c06-0f48-443f-ae03-43040b440a23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.872853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.872671    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbnm\" (UniqueName: \"kubernetes.io/projected/51807c06-0f48-443f-ae03-43040b440a23-kube-api-access-dfbnm\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.873481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.873416    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51807c06-0f48-443f-ae03-43040b440a23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.875206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.875151    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51807c06-0f48-443f-ae03-43040b440a23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.926633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.926598    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbnm\" (UniqueName: \"kubernetes.io/projected/51807c06-0f48-443f-ae03-43040b440a23-kube-api-access-dfbnm\") pod \"maas-api-revoke-sub-175f5c05-6f28p\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:53.945514 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:53.945481    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:54.007065 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:54.007035    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:54.323583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:54.323552    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"]
Nov 03 21:10:54.329300 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:10:54.329267    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91547460_ae34_4873_81d3_c18bc132a837.slice/crio-8004db53259a6fc5070a385963113a209ed4dac458699ff2f747e73d9d4eed19 WatchSource:0}: Error finding container 8004db53259a6fc5070a385963113a209ed4dac458699ff2f747e73d9d4eed19: Status 404 returned error can't find the container with id 8004db53259a6fc5070a385963113a209ed4dac458699ff2f747e73d9d4eed19
Nov 03 21:10:54.437203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:54.437166    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"]
Nov 03 21:10:54.443352 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:10:54.443325    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51807c06_0f48_443f_ae03_43040b440a23.slice/crio-213b9c6b7b8ed83b76aeda3418b4c319de95cdbde4282619560c62614d6a8886 WatchSource:0}: Error finding container 213b9c6b7b8ed83b76aeda3418b4c319de95cdbde4282619560c62614d6a8886: Status 404 returned error can't find the container with id 213b9c6b7b8ed83b76aeda3418b4c319de95cdbde4282619560c62614d6a8886
Nov 03 21:10:54.479936 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:54.479887    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:55.045222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.045136    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:10:55.079528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.079484    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p" event={"ID":"51807c06-0f48-443f-ae03-43040b440a23","Type":"ContainerStarted","Data":"9dc1ab8e59531f0f6b8365fbcf51d9657749552f98bdc40c1a13e7d75301e509"}
Nov 03 21:10:55.079709 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.079556    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p" event={"ID":"51807c06-0f48-443f-ae03-43040b440a23","Type":"ContainerStarted","Data":"213b9c6b7b8ed83b76aeda3418b4c319de95cdbde4282619560c62614d6a8886"}
Nov 03 21:10:55.081556 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.081529    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f" event={"ID":"91547460-ae34-4873-81d3-c18bc132a837","Type":"ContainerStarted","Data":"9bea1cb8574ce3f71c22f4b1e7e4e5706698bf1281cba14adb1b65a292002903"}
Nov 03 21:10:55.081684 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.081574    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f" event={"ID":"91547460-ae34-4873-81d3-c18bc132a837","Type":"ContainerStarted","Data":"8004db53259a6fc5070a385963113a209ed4dac458699ff2f747e73d9d4eed19"}
Nov 03 21:10:55.188933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.188869    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p" podStartSLOduration=2.188852843 podStartE2EDuration="2.188852843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:55.185854704 +0000 UTC m=+2495.189813428" watchObservedRunningTime="2026-11-03 21:10:55.188852843 +0000 UTC m=+2495.192811566"
Nov 03 21:10:55.294968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.294901    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f" podStartSLOduration=2.294882076 podStartE2EDuration="2.294882076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:55.292119045 +0000 UTC m=+2495.296077770" watchObservedRunningTime="2026-11-03 21:10:55.294882076 +0000 UTC m=+2495.298840801"
Nov 03 21:10:55.635460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.635431    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 21:10:55.708469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.708412    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istiod-ca-cert" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:55.708694 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.708669    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istiod-ca-cert\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.709600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.709164    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-token\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-token\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.709837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.709804    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-envoy\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.727463 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.727357    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-envoy" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:55.727633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.727606    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-certs\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.728027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.727991    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-socket" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:55.728574 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.728540    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-certs" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:55.728742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.728722    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-socket\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.728962 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.728910    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-credential-socket\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.729587 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.729389    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-credential-socket" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:55.729859 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.729841    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-data\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.730460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.730443    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-podinfo\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.734807 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.730225    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-data" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:55.735147 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.735085    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-kube-api-access-n6fwt\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-kube-api-access-n6fwt\") pod \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\" (UID: \"924eacd5-f3cd-47b4-8607-0f5dc26c4747\") "
Nov 03 21:10:55.736615 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.736596    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:55.736770 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.736757    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:55.736872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.736859    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:55.737023 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.737011    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:55.737133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.737123    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:55.737240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.737229    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:55.747825 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.747770    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-token" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:55.751350 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.751314    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-kube-api-access-n6fwt" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "kube-api-access-n6fwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:55.752494 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.752244    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-podinfo" pod "924eacd5-f3cd-47b4-8607-0f5dc26c4747" (UID: "924eacd5-f3cd-47b4-8607-0f5dc26c4747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:55.839394 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.839354    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:55.839394 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.839395    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6fwt\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-kube-api-access-n6fwt\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:55.839659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:55.839414    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924eacd5-f3cd-47b4-8607-0f5dc26c4747-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:56.092908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.092868    5196 generic.go:356] "Generic (PLEG): container finished" podID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerID="f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8" exitCode=0
Nov 03 21:10:56.093441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.092922    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"
Nov 03 21:10:56.093441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.092955    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" event={"ID":"924eacd5-f3cd-47b4-8607-0f5dc26c4747","Type":"ContainerDied","Data":"f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8"}
Nov 03 21:10:56.093441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.093013    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm" event={"ID":"924eacd5-f3cd-47b4-8607-0f5dc26c4747","Type":"ContainerDied","Data":"13e8a7e9a8a3a95ed4efeed1d0197f3cf28de6123e45b040029a11d7a864403b"}
Nov 03 21:10:56.093441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.093044    5196 scope.go:122] "RemoveContainer" containerID="f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8"
Nov 03 21:10:56.095133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.094864    5196 generic.go:356] "Generic (PLEG): container finished" podID="51807c06-0f48-443f-ae03-43040b440a23" containerID="9dc1ab8e59531f0f6b8365fbcf51d9657749552f98bdc40c1a13e7d75301e509" exitCode=0
Nov 03 21:10:56.095133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.094909    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p" event={"ID":"51807c06-0f48-443f-ae03-43040b440a23","Type":"ContainerDied","Data":"9dc1ab8e59531f0f6b8365fbcf51d9657749552f98bdc40c1a13e7d75301e509"}
Nov 03 21:10:56.097313 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.097268    5196 generic.go:356] "Generic (PLEG): container finished" podID="91547460-ae34-4873-81d3-c18bc132a837" containerID="9bea1cb8574ce3f71c22f4b1e7e4e5706698bf1281cba14adb1b65a292002903" exitCode=0
Nov 03 21:10:56.097447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.097321    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f" event={"ID":"91547460-ae34-4873-81d3-c18bc132a837","Type":"ContainerDied","Data":"9bea1cb8574ce3f71c22f4b1e7e4e5706698bf1281cba14adb1b65a292002903"}
Nov 03 21:10:56.106387 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.106356    5196 scope.go:122] "RemoveContainer" containerID="f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8"
Nov 03 21:10:56.111170 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:10:56.110553    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8\": container with ID starting with f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8 not found: ID does not exist" containerID="f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8"
Nov 03 21:10:56.111170 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.110607    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8"} err="failed to get container status \"f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8\": rpc error: code = NotFound desc = could not find container \"f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8\": container with ID starting with f2e2fec364909d4c264a1f9639e90da0262b64fa15cc3a0e3f5bb3289b6a92b8 not found: ID does not exist"
Nov 03 21:10:56.488399 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.487167    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"]
Nov 03 21:10:56.491243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.491211    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-2jjvm"]
Nov 03 21:10:56.662525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:56.662480    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:57.405864 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.405786    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:57.408712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.408676    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51807c06-0f48-443f-ae03-43040b440a23-kube-api-access-dfbnm\" (UniqueName: \"kubernetes.io/projected/51807c06-0f48-443f-ae03-43040b440a23-kube-api-access-dfbnm\") pod \"51807c06-0f48-443f-ae03-43040b440a23\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") "
Nov 03 21:10:57.408881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.408766    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51807c06-0f48-443f-ae03-43040b440a23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51807c06-0f48-443f-ae03-43040b440a23-maas-api-xks-ca\") pod \"51807c06-0f48-443f-ae03-43040b440a23\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") "
Nov 03 21:10:57.408881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.408868    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51807c06-0f48-443f-ae03-43040b440a23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51807c06-0f48-443f-ae03-43040b440a23-maas-api-openshift-ca\") pod \"51807c06-0f48-443f-ae03-43040b440a23\" (UID: \"51807c06-0f48-443f-ae03-43040b440a23\") "
Nov 03 21:10:57.409807 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.409780    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51807c06-0f48-443f-ae03-43040b440a23-maas-api-openshift-ca" pod "51807c06-0f48-443f-ae03-43040b440a23" (UID: "51807c06-0f48-443f-ae03-43040b440a23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:57.412538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.412497    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51807c06-0f48-443f-ae03-43040b440a23-kube-api-access-dfbnm" pod "51807c06-0f48-443f-ae03-43040b440a23" (UID: "51807c06-0f48-443f-ae03-43040b440a23"). InnerVolumeSpecName "kube-api-access-dfbnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:57.413144 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.413076    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51807c06-0f48-443f-ae03-43040b440a23-maas-api-xks-ca" pod "51807c06-0f48-443f-ae03-43040b440a23" (UID: "51807c06-0f48-443f-ae03-43040b440a23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:57.514733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.514326    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51807c06-0f48-443f-ae03-43040b440a23-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:57.514733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.514367    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbnm\" (UniqueName: \"kubernetes.io/projected/51807c06-0f48-443f-ae03-43040b440a23-kube-api-access-dfbnm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:57.514733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.514384    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51807c06-0f48-443f-ae03-43040b440a23-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:57.627439 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.627392    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:57.717643 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.717532    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91547460-ae34-4873-81d3-c18bc132a837-kube-api-access-jtz6f\" (UniqueName: \"kubernetes.io/projected/91547460-ae34-4873-81d3-c18bc132a837-kube-api-access-jtz6f\") pod \"91547460-ae34-4873-81d3-c18bc132a837\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") "
Nov 03 21:10:57.718247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.718212    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91547460-ae34-4873-81d3-c18bc132a837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91547460-ae34-4873-81d3-c18bc132a837-maas-api-xks-ca\") pod \"91547460-ae34-4873-81d3-c18bc132a837\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") "
Nov 03 21:10:57.718388 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.718270    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91547460-ae34-4873-81d3-c18bc132a837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91547460-ae34-4873-81d3-c18bc132a837-maas-api-openshift-ca\") pod \"91547460-ae34-4873-81d3-c18bc132a837\" (UID: \"91547460-ae34-4873-81d3-c18bc132a837\") "
Nov 03 21:10:57.718970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.718935    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91547460-ae34-4873-81d3-c18bc132a837-maas-api-openshift-ca" pod "91547460-ae34-4873-81d3-c18bc132a837" (UID: "91547460-ae34-4873-81d3-c18bc132a837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:57.720843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.720798    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91547460-ae34-4873-81d3-c18bc132a837-maas-api-xks-ca" pod "91547460-ae34-4873-81d3-c18bc132a837" (UID: "91547460-ae34-4873-81d3-c18bc132a837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:57.720961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.720939    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91547460-ae34-4873-81d3-c18bc132a837-kube-api-access-jtz6f" pod "91547460-ae34-4873-81d3-c18bc132a837" (UID: "91547460-ae34-4873-81d3-c18bc132a837"). InnerVolumeSpecName "kube-api-access-jtz6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:57.827282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.827195    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtz6f\" (UniqueName: \"kubernetes.io/projected/91547460-ae34-4873-81d3-c18bc132a837-kube-api-access-jtz6f\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:57.827282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.827235    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91547460-ae34-4873-81d3-c18bc132a837-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:57.827282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:57.827252    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91547460-ae34-4873-81d3-c18bc132a837-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:58.129665 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:58.129627    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" path="/var/lib/kubelet/pods/924eacd5-f3cd-47b4-8607-0f5dc26c4747/volumes"
Nov 03 21:10:58.136047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:58.135780    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f" event={"ID":"91547460-ae34-4873-81d3-c18bc132a837","Type":"ContainerDied","Data":"8004db53259a6fc5070a385963113a209ed4dac458699ff2f747e73d9d4eed19"}
Nov 03 21:10:58.136047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:58.135844    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8004db53259a6fc5070a385963113a209ed4dac458699ff2f747e73d9d4eed19"
Nov 03 21:10:58.136047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:58.135915    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"
Nov 03 21:10:58.142062 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:58.141801    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p" event={"ID":"51807c06-0f48-443f-ae03-43040b440a23","Type":"ContainerDied","Data":"213b9c6b7b8ed83b76aeda3418b4c319de95cdbde4282619560c62614d6a8886"}
Nov 03 21:10:58.142062 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:58.141860    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="213b9c6b7b8ed83b76aeda3418b4c319de95cdbde4282619560c62614d6a8886"
Nov 03 21:10:58.142062 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:58.141929    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"
Nov 03 21:10:58.831274 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:58.831230    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:59.706322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:59.706284    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"]
Nov 03 21:10:59.706790 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:59.706754    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy" containerID="cri-o://3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd" gracePeriod=30
Nov 03 21:10:59.950158 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:10:59.950119    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:00.120388 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:00.120076    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:00.347521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:00.347488    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:00.586720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:00.586671    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:00.745173 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:00.745140    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:00.831768 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:00.831725    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"]
Nov 03 21:11:00.877448 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:00.877357    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fd9afe-27cx2"]
Nov 03 21:11:01.277242 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:01.277150    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:02.109715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:02.109670    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09099f2-173b-49d3-b98f-e31aa94749d2" path="/var/lib/kubelet/pods/d09099f2-173b-49d3-b98f-e31aa94749d2/volumes"
Nov 03 21:11:02.347794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:02.347754    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:02.587189 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:02.587149    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:02.770946 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:02.770907    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:11:02.771156 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:02.770963    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:11:03.189081 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:03.189038    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"]
Nov 03 21:11:03.189367 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:03.189331    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy" containerID="cri-o://3e0923e377f8e9d020e347591da9f6fc99e9906aa38ce1275b6edbfb93000a06" gracePeriod=30
Nov 03 21:11:04.989785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:04.989706    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 21:11:05.105238 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105208    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-kube-api-access-4rgcr\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-kube-api-access-4rgcr\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105279    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ffa19a-6140-46db-b407-4be9e07da949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ffa19a-6140-46db-b407-4be9e07da949-istiod-ca-cert\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105311    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-socket\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105345    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1ffa19a-6140-46db-b407-4be9e07da949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ffa19a-6140-46db-b407-4be9e07da949-istio-podinfo\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105436    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-credential-socket\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105463    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-istio-token\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-istio-token\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105531    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-certs\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105552    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-data\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105605    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-envoy\") pod \"d1ffa19a-6140-46db-b407-4be9e07da949\" (UID: \"d1ffa19a-6140-46db-b407-4be9e07da949\") "
Nov 03 21:11:05.105860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105661    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-socket" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:05.105860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105677    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-credential-socket" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:05.105860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105747    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ffa19a-6140-46db-b407-4be9e07da949-istiod-ca-cert" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:11:05.105860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105844    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-certs" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:05.105860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.105855    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-data" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:05.106530 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.106183    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.106530 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.106203    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.106530 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.106213    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.106530 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.106225    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.106530 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.106238    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ffa19a-6140-46db-b407-4be9e07da949-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.107808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.107771    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1ffa19a-6140-46db-b407-4be9e07da949-istio-podinfo" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:11:05.107965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.107942    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-kube-api-access-4rgcr" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "kube-api-access-4rgcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:05.108401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.108377    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-envoy" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:05.108401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.108396    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-istio-token" pod "d1ffa19a-6140-46db-b407-4be9e07da949" (UID: "d1ffa19a-6140-46db-b407-4be9e07da949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:05.207311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.207270    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ffa19a-6140-46db-b407-4be9e07da949-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.207311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.207314    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rgcr\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-kube-api-access-4rgcr\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.207548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.207330    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ffa19a-6140-46db-b407-4be9e07da949-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.207548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.207343    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ffa19a-6140-46db-b407-4be9e07da949-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:05.277369 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.277268    5196 generic.go:356] "Generic (PLEG): container finished" podID="d1ffa19a-6140-46db-b407-4be9e07da949" containerID="3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd" exitCode=0
Nov 03 21:11:05.277369 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.277328    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"
Nov 03 21:11:05.277369 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.277349    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" event={"ID":"d1ffa19a-6140-46db-b407-4be9e07da949","Type":"ContainerDied","Data":"3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd"}
Nov 03 21:11:05.277628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.277413    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh" event={"ID":"d1ffa19a-6140-46db-b407-4be9e07da949","Type":"ContainerDied","Data":"b4271c2332774237ead504ecbf7fdc1b9d8c0166c8f61d7aea16437ae7584f0a"}
Nov 03 21:11:05.277628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.277435    5196 scope.go:122] "RemoveContainer" containerID="3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd"
Nov 03 21:11:05.290201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.290177    5196 scope.go:122] "RemoveContainer" containerID="3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd"
Nov 03 21:11:05.290551 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:11:05.290531    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd\": container with ID starting with 3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd not found: ID does not exist" containerID="3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd"
Nov 03 21:11:05.290638 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.290561    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd"} err="failed to get container status \"3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd\": rpc error: code = NotFound desc = could not find container \"3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd\": container with ID starting with 3a94fb419d9dfc36bd6bdb4fecce6367055372c1f0f8672033ce0d68456bc6dd not found: ID does not exist"
Nov 03 21:11:05.434794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.434757    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"]
Nov 03 21:11:05.438748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.438716    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-mvdnh"]
Nov 03 21:11:05.761612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:05.761558    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:06.212618 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:06.212576    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" path="/var/lib/kubelet/pods/d1ffa19a-6140-46db-b407-4be9e07da949/volumes"
Nov 03 21:11:08.121845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.121806    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:08.323009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.322968    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:08.355405 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.355372    5196 generic.go:356] "Generic (PLEG): container finished" podID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerID="3e0923e377f8e9d020e347591da9f6fc99e9906aa38ce1275b6edbfb93000a06" exitCode=0
Nov 03 21:11:08.355597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.355446    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" event={"ID":"207d53e6-6438-445a-a9c5-eda28ffed68f","Type":"ContainerDied","Data":"3e0923e377f8e9d020e347591da9f6fc99e9906aa38ce1275b6edbfb93000a06"}
Nov 03 21:11:08.433441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.433331    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"]
Nov 03 21:11:08.433677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.433649    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy" containerID="cri-o://c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d" gracePeriod=30
Nov 03 21:11:08.481529 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.481495    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:08.489480 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.489450    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 21:11:08.593706 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.593656    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-podinfo\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.593923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.593764    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-kube-api-access-lvzpv\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-kube-api-access-lvzpv\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.593923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.593828    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207d53e6-6438-445a-a9c5-eda28ffed68f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207d53e6-6438-445a-a9c5-eda28ffed68f-istiod-ca-cert\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.593923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.593887    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-envoy\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.594150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.593930    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-data\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.594150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.593974    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-token\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-token\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.594150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.594046    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-certs\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.594150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.594107    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-socket\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.594150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.594132    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-credential-socket\") pod \"207d53e6-6438-445a-a9c5-eda28ffed68f\" (UID: \"207d53e6-6438-445a-a9c5-eda28ffed68f\") "
Nov 03 21:11:08.594769 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.594738    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-certs" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:08.594930 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.594906    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-socket" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:08.595240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.594989    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:08.595240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.594997    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-data" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:08.595240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.595007    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207d53e6-6438-445a-a9c5-eda28ffed68f-istiod-ca-cert" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:11:08.595240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.595060    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-credential-socket" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:08.602171 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.597269    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-podinfo" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:11:08.602171 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.597645    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-token" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:08.602171 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.598275    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-kube-api-access-lvzpv" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "kube-api-access-lvzpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:08.602171 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.600490    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-envoy" pod "207d53e6-6438-445a-a9c5-eda28ffed68f" (UID: "207d53e6-6438-445a-a9c5-eda28ffed68f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:08.695873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.695798    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:08.695873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.695827    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvzpv\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-kube-api-access-lvzpv\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:08.695873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.695839    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207d53e6-6438-445a-a9c5-eda28ffed68f-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:08.695873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.695849    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:08.695873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.695857    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:08.695873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.695865    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207d53e6-6438-445a-a9c5-eda28ffed68f-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:08.695873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.695876    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:08.696283 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:08.695884    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207d53e6-6438-445a-a9c5-eda28ffed68f-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:09.365585 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:09.365555    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"
Nov 03 21:11:09.365585 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:09.365574    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn" event={"ID":"207d53e6-6438-445a-a9c5-eda28ffed68f","Type":"ContainerDied","Data":"c6d9a1ade090ea6961544eaa38f6aed0b672545460d6b52cca7b0f34a7c4b8fc"}
Nov 03 21:11:09.366179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:09.365644    5196 scope.go:122] "RemoveContainer" containerID="3e0923e377f8e9d020e347591da9f6fc99e9906aa38ce1275b6edbfb93000a06"
Nov 03 21:11:09.570891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:09.570854    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"]
Nov 03 21:11:09.618240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:09.618131    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-dhrpn"]
Nov 03 21:11:10.109739 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:10.109707    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" path="/var/lib/kubelet/pods/207d53e6-6438-445a-a9c5-eda28ffed68f/volumes"
Nov 03 21:11:10.555679 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:10.555597    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:13.717221 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.717160    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 21:11:13.759513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759477    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-kube-api-access-q9chb\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-kube-api-access-q9chb\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.759713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759549    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-podinfo\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.759713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759601    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-certs\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.759713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759632    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istiod-ca-cert\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.759713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759692    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-socket\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.759957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759718    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-credential-socket\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.759957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759737    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-envoy\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.759957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759784    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-data\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.759957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759811    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-token\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-token\") pod \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\" (UID: \"fa307edc-bb9a-4d62-9c3b-17388df1bab9\") "
Nov 03 21:11:13.760233 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.759979    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-certs" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:13.760233 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.760071    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-socket" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:13.760340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.760244    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-data" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:13.760340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.760314    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istiod-ca-cert" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:11:13.760629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.760549    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:13.760629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.760570    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:13.760629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.760583    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:13.760629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.760593    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:13.760857 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.760637    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-credential-socket" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:13.762185 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.762162    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-podinfo" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:11:13.762309 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.762238    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-kube-api-access-q9chb" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "kube-api-access-q9chb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:13.762657 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.762629    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-envoy" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:13.762657 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.762647    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-token" pod "fa307edc-bb9a-4d62-9c3b-17388df1bab9" (UID: "fa307edc-bb9a-4d62-9c3b-17388df1bab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:13.861806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.861769    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:13.861806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.861807    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:13.862037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.861821    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:13.862037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.861835    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9chb\" (UniqueName: \"kubernetes.io/projected/fa307edc-bb9a-4d62-9c3b-17388df1bab9-kube-api-access-q9chb\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:13.862037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:13.861846    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa307edc-bb9a-4d62-9c3b-17388df1bab9-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:14.409351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.409239    5196 generic.go:356] "Generic (PLEG): container finished" podID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerID="c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d" exitCode=0
Nov 03 21:11:14.409351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.409317    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"
Nov 03 21:11:14.409351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.409327    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" event={"ID":"fa307edc-bb9a-4d62-9c3b-17388df1bab9","Type":"ContainerDied","Data":"c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d"}
Nov 03 21:11:14.409624 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.409389    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh" event={"ID":"fa307edc-bb9a-4d62-9c3b-17388df1bab9","Type":"ContainerDied","Data":"f5d736f17dfc5f916cc765028db9227deaf25edeaa5c9c5599de4821c40fcd42"}
Nov 03 21:11:14.409624 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.409411    5196 scope.go:122] "RemoveContainer" containerID="c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d"
Nov 03 21:11:14.421647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.421628    5196 scope.go:122] "RemoveContainer" containerID="c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d"
Nov 03 21:11:14.422028 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:11:14.422005    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d\": container with ID starting with c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d not found: ID does not exist" containerID="c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d"
Nov 03 21:11:14.422209 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.422036    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d"} err="failed to get container status \"c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d\": rpc error: code = NotFound desc = could not find container \"c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d\": container with ID starting with c9837c627df38d99d1d89d8d48fa295b32ef3e25ad2bdbe57c07eb16aa85b89d not found: ID does not exist"
Nov 03 21:11:14.566056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.566019    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"]
Nov 03 21:11:14.569532 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:14.569503    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fd9afe-openshift-default-84646db8cf-cm7hh"]
Nov 03 21:11:15.252505 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:15.252471    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:16.109590 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:16.109553    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" path="/var/lib/kubelet/pods/fa307edc-bb9a-4d62-9c3b-17388df1bab9/volumes"
Nov 03 21:11:18.485551 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:11:18.485515    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:11:19.960398 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:19.960361    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:24.677154 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:24.676998    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:26.520813 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:26.520780    5196 scope.go:122] "RemoveContainer" containerID="7ce56764db8b39653c5f01bd333774ca62420afc85666b0c57405b5f40f70257"
Nov 03 21:11:29.280274 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280242    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"]
Nov 03 21:11:29.280872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280858    5196 cpu_manager.go:414] "containerMap: removing container" podUID="51807c06-0f48-443f-ae03-43040b440a23" containerName="revoke-keys"
Nov 03 21:11:29.280919 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280872    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51807c06-0f48-443f-ae03-43040b440a23" containerName="revoke-keys"
Nov 03 21:11:29.280919 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280890    5196 cpu_manager.go:414] "containerMap: removing container" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy"
Nov 03 21:11:29.280919 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280896    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy"
Nov 03 21:11:29.281020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280925    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy"
Nov 03 21:11:29.281020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280932    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy"
Nov 03 21:11:29.281020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280945    5196 cpu_manager.go:414] "containerMap: removing container" podUID="91547460-ae34-4873-81d3-c18bc132a837" containerName="revoke-keys"
Nov 03 21:11:29.281020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280953    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91547460-ae34-4873-81d3-c18bc132a837" containerName="revoke-keys"
Nov 03 21:11:29.281020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280969    5196 cpu_manager.go:414] "containerMap: removing container" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy"
Nov 03 21:11:29.281020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280973    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy"
Nov 03 21:11:29.281020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280989    5196 cpu_manager.go:414] "containerMap: removing container" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy"
Nov 03 21:11:29.281020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.280994    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy"
Nov 03 21:11:29.281305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.281111    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ffa19a-6140-46db-b407-4be9e07da949" containerName="istio-proxy"
Nov 03 21:11:29.281305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.281120    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="91547460-ae34-4873-81d3-c18bc132a837" containerName="revoke-keys"
Nov 03 21:11:29.281305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.281129    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="207d53e6-6438-445a-a9c5-eda28ffed68f" containerName="istio-proxy"
Nov 03 21:11:29.281305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.281138    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="924eacd5-f3cd-47b4-8607-0f5dc26c4747" containerName="istio-proxy"
Nov 03 21:11:29.281305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.281145    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="51807c06-0f48-443f-ae03-43040b440a23" containerName="revoke-keys"
Nov 03 21:11:29.281305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.281151    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa307edc-bb9a-4d62-9c3b-17388df1bab9" containerName="istio-proxy"
Nov 03 21:11:29.286782 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.286760    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.336591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.336557    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7c34ff\"/\"kube-root-ca.crt\""
Nov 03 21:11:29.336591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.336564    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7c34ff\"/\"openshift-service-ca.crt\""
Nov 03 21:11:29.345752 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.345716    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:29.352041 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.352012    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7c34ff\"/\"default-dockercfg-cvx9q\""
Nov 03 21:11:29.352193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.352043    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7c34ff\"/\"test-model-7c34ff-kserve-self-signed-certs\""
Nov 03 21:11:29.390287 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.390245    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"]
Nov 03 21:11:29.457243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.457212    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-dshm\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.457243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.457258    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-model-cache\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.457506 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.457279    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tmp-dir\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.457506 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.457445    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jdx\" (UniqueName: \"kubernetes.io/projected/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-kube-api-access-x2jdx\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.457506 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.457476    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tls-certs\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.457506 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.457500    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-home\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.489288 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.489244    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:29.558605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.558509    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-dshm\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.558605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.558553    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-model-cache\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.558605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.558569    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tmp-dir\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.558873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.558633    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jdx\" (UniqueName: \"kubernetes.io/projected/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-kube-api-access-x2jdx\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.558873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.558654    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tls-certs\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.558873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.558675    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-home\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.559065 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.559044    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tmp-dir\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.559193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.559128    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-model-cache\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.559193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.559171    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-home\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.561134 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.561068    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-dshm\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.561364 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.561345    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tls-certs\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.612726 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.612685    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jdx\" (UniqueName: \"kubernetes.io/projected/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-kube-api-access-x2jdx\") pod \"test-model-7c34ff-kserve-c4d79d5f7-k8qsx\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") " pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:29.900784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:29.900750    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:30.282374 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:30.282287    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"]
Nov 03 21:11:30.293393 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:11:30.293356    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb9bd17_20c5_428f_bb2f_e222f9c5e1e7.slice/crio-c3ab961269c63a07f50e0d34270870777a83c013d34b5e3df3de3ec9183d4ada WatchSource:0}: Error finding container c3ab961269c63a07f50e0d34270870777a83c013d34b5e3df3de3ec9183d4ada: Status 404 returned error can't find the container with id c3ab961269c63a07f50e0d34270870777a83c013d34b5e3df3de3ec9183d4ada
Nov 03 21:11:30.545612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:30.545505    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx" event={"ID":"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7","Type":"ContainerStarted","Data":"515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55"}
Nov 03 21:11:30.545612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:30.545557    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx" event={"ID":"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7","Type":"ContainerStarted","Data":"c3ab961269c63a07f50e0d34270870777a83c013d34b5e3df3de3ec9183d4ada"}
Nov 03 21:11:30.653745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:30.653690    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx" podStartSLOduration=1.653671724 podStartE2EDuration="1.653671724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:30.653219798 +0000 UTC m=+2530.657178541" watchObservedRunningTime="2026-11-03 21:11:30.653671724 +0000 UTC m=+2530.657630483"
Nov 03 21:11:30.900894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:30.900857    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:32.321031 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.320995    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"]
Nov 03 21:11:32.325545 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.325522    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.379774 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.379737    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"]
Nov 03 21:11:32.420888 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.420849    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-98d1d2\"/\"default-dockercfg-ml2gw\""
Nov 03 21:11:32.420888 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.420856    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-98d1d2\"/\"openshift-service-ca.crt\""
Nov 03 21:11:32.421131 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.420903    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-98d1d2\"/\"kube-root-ca.crt\""
Nov 03 21:11:32.422389 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.422372    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-98d1d2\"/\"test-model-98d1d2-kserve-self-signed-certs\""
Nov 03 21:11:32.442689 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.442657    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:32.495814 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.495778    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-model-cache\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.496011 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.495921    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tls-certs\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.496011 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.495944    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74pzd\" (UniqueName: \"kubernetes.io/projected/8fd3fbdc-5419-4b73-84b9-375fc52ace96-kube-api-access-74pzd\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.496104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.496043    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-home\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.496104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.496075    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-dshm\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.496251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.496235    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tmp-dir\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.597669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.597556    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tls-certs\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.597669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.597609    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74pzd\" (UniqueName: \"kubernetes.io/projected/8fd3fbdc-5419-4b73-84b9-375fc52ace96-kube-api-access-74pzd\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.597927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.597683    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-home\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.597927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.597706    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-dshm\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.597927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.597794    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tmp-dir\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.597927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.597881    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-model-cache\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.598229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.598204    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-home\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.598285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.598232    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tmp-dir\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.598343 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.598324    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-model-cache\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.600333 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.600314    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-dshm\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.600594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.600573    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tls-certs\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.651695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.651662    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74pzd\" (UniqueName: \"kubernetes.io/projected/8fd3fbdc-5419-4b73-84b9-375fc52ace96-kube-api-access-74pzd\") pod \"test-model-98d1d2-kserve-7c79bfc655-4qz4k\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") " pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:32.940688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:32.940590    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:33.320803 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:33.320759    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"]
Nov 03 21:11:33.326954 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:11:33.326923    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fd3fbdc_5419_4b73_84b9_375fc52ace96.slice/crio-63907df8b796e0f53f3fff71e574b855a76b68b7a6124e652887132616aeac67 WatchSource:0}: Error finding container 63907df8b796e0f53f3fff71e574b855a76b68b7a6124e652887132616aeac67: Status 404 returned error can't find the container with id 63907df8b796e0f53f3fff71e574b855a76b68b7a6124e652887132616aeac67
Nov 03 21:11:33.574418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:33.574334    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k" event={"ID":"8fd3fbdc-5419-4b73-84b9-375fc52ace96","Type":"ContainerStarted","Data":"b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392"}
Nov 03 21:11:33.574418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:33.574382    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k" event={"ID":"8fd3fbdc-5419-4b73-84b9-375fc52ace96","Type":"ContainerStarted","Data":"63907df8b796e0f53f3fff71e574b855a76b68b7a6124e652887132616aeac67"}
Nov 03 21:11:33.682215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:33.682143    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k" podStartSLOduration=1.682079237 podStartE2EDuration="1.682079237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:33.681356358 +0000 UTC m=+2533.685315095" watchObservedRunningTime="2026-11-03 21:11:33.682079237 +0000 UTC m=+2533.686038070"
Nov 03 21:11:33.940910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:33.940830    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:34.050231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:34.050195    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:38.853155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:38.853119    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:39.901064 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:39.901025    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:39.915620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:39.915593    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:40.657829 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:40.657798    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:11:42.941773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:42.941737    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:42.955639 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:42.955611    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:43.664297 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:43.664242    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:43.681523 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:43.681486    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:11:47.774931 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:47.774891    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:48.485892 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:11:48.485851    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:11:49.263489 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:49.263456    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:49.490033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:49.489976    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:49.634079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:49.634044    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:49.747901 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:49.747867    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"]
Nov 03 21:11:49.794894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:49.794859    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b119f2-lq8k5"]
Nov 03 21:11:50.110650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:50.110617    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b009d39-60db-49fe-8f0e-5953cdf37b17" path="/var/lib/kubelet/pods/1b009d39-60db-49fe-8f0e-5953cdf37b17/volumes"
Nov 03 21:11:50.127212 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:50.127172    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:50.284141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:50.284109    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:50.438270 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:50.438146    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:52.224489 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.224453    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"]
Nov 03 21:11:52.229319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.229292    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.280669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.280639    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"]
Nov 03 21:11:52.323274 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.323241    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:11:52.323274 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.323261    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:11:52.383990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.383952    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.384229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.384152    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjkxl\" (UniqueName: \"kubernetes.io/projected/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-kube-api-access-qjkxl\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.384229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.384203    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.485178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.485036    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.485382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.485245    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjkxl\" (UniqueName: \"kubernetes.io/projected/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-kube-api-access-qjkxl\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.485382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.485311    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.485772 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.485747    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.487804 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.487779    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.539322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.539284    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjkxl\" (UniqueName: \"kubernetes.io/projected/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-kube-api-access-qjkxl\") pod \"maas-api-revoke-sub-9a637bba-l6bpb\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.544105 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.544067    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:52.922307 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:52.922272    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"]
Nov 03 21:11:52.928483 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:11:52.928446    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba23cc96_8ad1_4ddc_bc1e_3578bb73afb7.slice/crio-88f93add24ac2575f1dd23c6d63d83a60d527b64d609014f2b9c2ef8d82c3ab6 WatchSource:0}: Error finding container 88f93add24ac2575f1dd23c6d63d83a60d527b64d609014f2b9c2ef8d82c3ab6: Status 404 returned error can't find the container with id 88f93add24ac2575f1dd23c6d63d83a60d527b64d609014f2b9c2ef8d82c3ab6
Nov 03 21:11:53.755661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:53.755618    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb" event={"ID":"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7","Type":"ContainerStarted","Data":"db0c930d98a5db4f263c2ae6da2b14d34dab2534c978314531961d1a19a79eb8"}
Nov 03 21:11:53.756075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:53.755687    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb" event={"ID":"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7","Type":"ContainerStarted","Data":"88f93add24ac2575f1dd23c6d63d83a60d527b64d609014f2b9c2ef8d82c3ab6"}
Nov 03 21:11:53.867113 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:53.867049    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb" podStartSLOduration=1.867033675 podStartE2EDuration="1.867033675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:53.866878947 +0000 UTC m=+2553.870837672" watchObservedRunningTime="2026-11-03 21:11:53.867033675 +0000 UTC m=+2553.870992399"
Nov 03 21:11:54.532958 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:54.532926    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:54.765630 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:54.765598    5196 generic.go:356] "Generic (PLEG): container finished" podID="ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7" containerID="db0c930d98a5db4f263c2ae6da2b14d34dab2534c978314531961d1a19a79eb8" exitCode=0
Nov 03 21:11:54.766159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:54.765696    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb" event={"ID":"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7","Type":"ContainerDied","Data":"db0c930d98a5db4f263c2ae6da2b14d34dab2534c978314531961d1a19a79eb8"}
Nov 03 21:11:55.942374 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:55.942341    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:56.040366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.040277    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-openshift-ca\") pod \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") "
Nov 03 21:11:56.040366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.040314    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-xks-ca\") pod \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") "
Nov 03 21:11:56.040631 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.040474    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-kube-api-access-qjkxl\" (UniqueName: \"kubernetes.io/projected/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-kube-api-access-qjkxl\") pod \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\" (UID: \"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7\") "
Nov 03 21:11:56.040726 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.040701    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-openshift-ca" pod "ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7" (UID: "ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:11:56.040891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.040874    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:56.042740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.042716    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-xks-ca" pod "ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7" (UID: "ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:11:56.042903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.042801    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-kube-api-access-qjkxl" pod "ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7" (UID: "ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7"). InnerVolumeSpecName "kube-api-access-qjkxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:56.142426 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.142392    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjkxl\" (UniqueName: \"kubernetes.io/projected/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-kube-api-access-qjkxl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:56.142426 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.142428    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:56.234363 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.234328    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:56.428788 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.428746    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:56.519765 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.519730    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"]
Nov 03 21:11:56.519984 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.519963    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy" containerID="cri-o://b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0" gracePeriod=30
Nov 03 21:11:56.584268 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.584233    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:56.783623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.783540    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"
Nov 03 21:11:56.783623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.783560    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb" event={"ID":"ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7","Type":"ContainerDied","Data":"88f93add24ac2575f1dd23c6d63d83a60d527b64d609014f2b9c2ef8d82c3ab6"}
Nov 03 21:11:56.783623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:56.783606    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88f93add24ac2575f1dd23c6d63d83a60d527b64d609014f2b9c2ef8d82c3ab6"
Nov 03 21:11:59.227903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:11:59.227870    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:01.813032 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.813000    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 21:12:01.826303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.826275    5196 generic.go:356] "Generic (PLEG): container finished" podID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerID="b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0" exitCode=0
Nov 03 21:12:01.826469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.826329    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"
Nov 03 21:12:01.826469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.826358    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" event={"ID":"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe","Type":"ContainerDied","Data":"b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0"}
Nov 03 21:12:01.826469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.826416    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb" event={"ID":"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe","Type":"ContainerDied","Data":"9a77f3826b49283681773290832b2f6f296db04a1235aec8d42662dd9f4d5368"}
Nov 03 21:12:01.826469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.826440    5196 scope.go:122] "RemoveContainer" containerID="b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0"
Nov 03 21:12:01.838333 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.838306    5196 scope.go:122] "RemoveContainer" containerID="b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0"
Nov 03 21:12:01.838637 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:12:01.838609    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0\": container with ID starting with b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0 not found: ID does not exist" containerID="b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0"
Nov 03 21:12:01.838720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.838651    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0"} err="failed to get container status \"b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0\": rpc error: code = NotFound desc = could not find container \"b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0\": container with ID starting with b1675bf5923a266f8d8dc89bfa64ce20b9172f9a15c6984a37ce794442aea0b0 not found: ID does not exist"
Nov 03 21:12:01.914333 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914300    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-kube-api-access-kzz5v\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-kube-api-access-kzz5v\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.914527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914351    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-socket\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.914527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914389    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-credential-socket\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.914629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914561    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-podinfo\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.914629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914616    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-envoy\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.914751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914653    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-socket" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:01.914751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914671    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-credential-socket" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:01.914751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914736    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istiod-ca-cert\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.914932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914911    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-certs\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.915004 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.914945    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-token\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-token\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.915063 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915004    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-data\") pod \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\" (UID: \"af5e55a3-0cc1-48c1-a42e-700f06a2e0fe\") "
Nov 03 21:12:01.915063 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915013    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istiod-ca-cert" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:01.915229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915164    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-certs" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:01.915468 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915322    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-data" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:01.915852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915825    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:01.915968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915855    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:01.915968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915870    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:01.915968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915885    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:01.915968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.915899    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:01.916945 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.916922    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-podinfo" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:12:01.917047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.916995    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-kube-api-access-kzz5v" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "kube-api-access-kzz5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:01.917543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.917526    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-envoy" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:01.917600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:01.917560    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-token" pod "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" (UID: "af5e55a3-0cc1-48c1-a42e-700f06a2e0fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:02.017753 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:02.017719    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzz5v\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-kube-api-access-kzz5v\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:02.017753 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:02.017750    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:02.017753 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:02.017761    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:02.018029 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:02.017770    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:02.236692 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:02.236651    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"]
Nov 03 21:12:02.240815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:02.240786    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b119f2-openshift-default-86cc8c8c67-flpkb"]
Nov 03 21:12:02.898003 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:02.897952    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:03.346024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.345975    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"]
Nov 03 21:12:03.347117 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.347065    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7" containerName="revoke-keys"
Nov 03 21:12:03.347117 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.347115    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7" containerName="revoke-keys"
Nov 03 21:12:03.347231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.347141    5196 cpu_manager.go:414] "containerMap: removing container" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy"
Nov 03 21:12:03.347231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.347149    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy"
Nov 03 21:12:03.347357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.347344    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7" containerName="revoke-keys"
Nov 03 21:12:03.347394 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.347367    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" containerName="istio-proxy"
Nov 03 21:12:03.352955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.352933    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.400664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.400631    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"]
Nov 03 21:12:03.437916 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.437875    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcnv\" (UniqueName: \"kubernetes.io/projected/e446063e-9950-4ca5-83fc-ef2920a151d2-kube-api-access-txcnv\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.438144 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.437998    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.438235 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.438217    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.443024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.442988    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:12:03.443024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.442999    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:12:03.539548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.539511    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.539740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.539574    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcnv\" (UniqueName: \"kubernetes.io/projected/e446063e-9950-4ca5-83fc-ef2920a151d2-kube-api-access-txcnv\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.539740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.539627    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.540310 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.540280    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.542085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.542057    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.593471 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.593433    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcnv\" (UniqueName: \"kubernetes.io/projected/e446063e-9950-4ca5-83fc-ef2920a151d2-kube-api-access-txcnv\") pod \"maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:03.668765 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:03.668683    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:04.057837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:04.057738    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"]
Nov 03 21:12:04.063897 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:12:04.063855    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode446063e_9950_4ca5_83fc_ef2920a151d2.slice/crio-e42f77a9691bbf8bbc30828a24adbf8889fd5a3dd663e4d821b3a271021935de WatchSource:0}: Error finding container e42f77a9691bbf8bbc30828a24adbf8889fd5a3dd663e4d821b3a271021935de: Status 404 returned error can't find the container with id e42f77a9691bbf8bbc30828a24adbf8889fd5a3dd663e4d821b3a271021935de
Nov 03 21:12:04.065147 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:04.065122    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:04.111830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:04.111798    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5e55a3-0cc1-48c1-a42e-700f06a2e0fe" path="/var/lib/kubelet/pods/af5e55a3-0cc1-48c1-a42e-700f06a2e0fe/volumes"
Nov 03 21:12:04.856493 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:04.856453    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx" event={"ID":"e446063e-9950-4ca5-83fc-ef2920a151d2","Type":"ContainerStarted","Data":"26b6ce8239ba350f72376285dffbadb6e36feb9af9317c76ba68d205bd68385f"}
Nov 03 21:12:04.856712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:04.856504    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx" event={"ID":"e446063e-9950-4ca5-83fc-ef2920a151d2","Type":"ContainerStarted","Data":"e42f77a9691bbf8bbc30828a24adbf8889fd5a3dd663e4d821b3a271021935de"}
Nov 03 21:12:04.966684 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:04.966629    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx" podStartSLOduration=1.9666139089999999 podStartE2EDuration="1.966613909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:04.962801391 +0000 UTC m=+2564.966760128" watchObservedRunningTime="2026-11-03 21:12:04.966613909 +0000 UTC m=+2564.970572632"
Nov 03 21:12:05.875271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:05.875227    5196 generic.go:356] "Generic (PLEG): container finished" podID="e446063e-9950-4ca5-83fc-ef2920a151d2" containerID="26b6ce8239ba350f72376285dffbadb6e36feb9af9317c76ba68d205bd68385f" exitCode=0
Nov 03 21:12:05.875716 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:05.875309    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx" event={"ID":"e446063e-9950-4ca5-83fc-ef2920a151d2","Type":"ContainerDied","Data":"26b6ce8239ba350f72376285dffbadb6e36feb9af9317c76ba68d205bd68385f"}
Nov 03 21:12:06.480452 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:06.480413    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:12:07.057835 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.057809    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:07.187691 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.187596    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-openshift-ca\") pod \"e446063e-9950-4ca5-83fc-ef2920a151d2\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") "
Nov 03 21:12:07.187906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.187680    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-xks-ca\") pod \"e446063e-9950-4ca5-83fc-ef2920a151d2\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") "
Nov 03 21:12:07.187970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.187910    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e446063e-9950-4ca5-83fc-ef2920a151d2-kube-api-access-txcnv\" (UniqueName: \"kubernetes.io/projected/e446063e-9950-4ca5-83fc-ef2920a151d2-kube-api-access-txcnv\") pod \"e446063e-9950-4ca5-83fc-ef2920a151d2\" (UID: \"e446063e-9950-4ca5-83fc-ef2920a151d2\") "
Nov 03 21:12:07.189669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.189639    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-openshift-ca" pod "e446063e-9950-4ca5-83fc-ef2920a151d2" (UID: "e446063e-9950-4ca5-83fc-ef2920a151d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:07.192500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.192466    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-xks-ca" pod "e446063e-9950-4ca5-83fc-ef2920a151d2" (UID: "e446063e-9950-4ca5-83fc-ef2920a151d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:07.193412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.193379    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e446063e-9950-4ca5-83fc-ef2920a151d2-kube-api-access-txcnv" pod "e446063e-9950-4ca5-83fc-ef2920a151d2" (UID: "e446063e-9950-4ca5-83fc-ef2920a151d2"). InnerVolumeSpecName "kube-api-access-txcnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:07.290459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.290417    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:07.290459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.290450    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txcnv\" (UniqueName: \"kubernetes.io/projected/e446063e-9950-4ca5-83fc-ef2920a151d2-kube-api-access-txcnv\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:07.290459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.290462    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e446063e-9950-4ca5-83fc-ef2920a151d2-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:07.835478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.835443    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"]
Nov 03 21:12:07.882651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.882617    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"]
Nov 03 21:12:07.886809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.886780    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"]
Nov 03 21:12:07.895045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.895008    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx" event={"ID":"e446063e-9950-4ca5-83fc-ef2920a151d2","Type":"ContainerDied","Data":"e42f77a9691bbf8bbc30828a24adbf8889fd5a3dd663e4d821b3a271021935de"}
Nov 03 21:12:07.895237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.895074    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42f77a9691bbf8bbc30828a24adbf8889fd5a3dd663e4d821b3a271021935de"
Nov 03 21:12:07.895237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:07.895160    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"
Nov 03 21:12:08.028662 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:08.028608    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:09.362216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:09.362177    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:14.129925 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:14.129880    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:14.856518 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:14.856482    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"]
Nov 03 21:12:14.962604 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:14.962568    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"]
Nov 03 21:12:15.239827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:15.239741    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:15.390261 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:15.390223    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:18.486044 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:12:18.486003    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:12:18.775244 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:18.775153    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:19.857157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:19.856913    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g" podUID="e43f5087-4ed9-4e95-bf72-57432ed7681c" containerName="payload-processing" containerID="cri-o://7ebb7666397c9416561479b2e12563ad152873ac8025dbee2777a3b0f9b5b383" gracePeriod=25
Nov 03 21:12:19.892959 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:12:19.892931    5196 logging.go:55] [core] [Channel #9490 SubChannel #9492]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.79:50052", ServerName: "10.132.0.79:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.79:50052: connect: connection refused"
Nov 03 21:12:19.963661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:19.963604    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8" podUID="b6df122a-3baa-4fba-b3b0-4e6521e682d8" containerName="payload-pre-processing" containerID="cri-o://359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a" gracePeriod=25
Nov 03 21:12:19.998824 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:19.998790    5196 generic.go:356] "Generic (PLEG): container finished" podID="e43f5087-4ed9-4e95-bf72-57432ed7681c" containerID="7ebb7666397c9416561479b2e12563ad152873ac8025dbee2777a3b0f9b5b383" exitCode=0
Nov 03 21:12:19.998957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:19.998847    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g" event={"ID":"e43f5087-4ed9-4e95-bf72-57432ed7681c","Type":"ContainerDied","Data":"7ebb7666397c9416561479b2e12563ad152873ac8025dbee2777a3b0f9b5b383"}
Nov 03 21:12:20.272012 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.271978    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 21:12:20.301382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.301351    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 21:12:20.466833 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.466748    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43f5087-4ed9-4e95-bf72-57432ed7681c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43f5087-4ed9-4e95-bf72-57432ed7681c-plugins-config-volume\") pod \"e43f5087-4ed9-4e95-bf72-57432ed7681c\" (UID: \"e43f5087-4ed9-4e95-bf72-57432ed7681c\") "
Nov 03 21:12:20.466990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.466904    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43f5087-4ed9-4e95-bf72-57432ed7681c-kube-api-access-v5crq\" (UniqueName: \"kubernetes.io/projected/e43f5087-4ed9-4e95-bf72-57432ed7681c-kube-api-access-v5crq\") pod \"e43f5087-4ed9-4e95-bf72-57432ed7681c\" (UID: \"e43f5087-4ed9-4e95-bf72-57432ed7681c\") "
Nov 03 21:12:20.466990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.466928    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6df122a-3baa-4fba-b3b0-4e6521e682d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6df122a-3baa-4fba-b3b0-4e6521e682d8-plugins-config-volume\") pod \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\" (UID: \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\") "
Nov 03 21:12:20.466990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.466961    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6df122a-3baa-4fba-b3b0-4e6521e682d8-kube-api-access-6mwrj\" (UniqueName: \"kubernetes.io/projected/b6df122a-3baa-4fba-b3b0-4e6521e682d8-kube-api-access-6mwrj\") pod \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\" (UID: \"b6df122a-3baa-4fba-b3b0-4e6521e682d8\") "
Nov 03 21:12:20.467192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.467165    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43f5087-4ed9-4e95-bf72-57432ed7681c-plugins-config-volume" pod "e43f5087-4ed9-4e95-bf72-57432ed7681c" (UID: "e43f5087-4ed9-4e95-bf72-57432ed7681c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:20.467412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.467389    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6df122a-3baa-4fba-b3b0-4e6521e682d8-plugins-config-volume" pod "b6df122a-3baa-4fba-b3b0-4e6521e682d8" (UID: "b6df122a-3baa-4fba-b3b0-4e6521e682d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:20.467575 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.467559    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6df122a-3baa-4fba-b3b0-4e6521e682d8-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:20.467725 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.467576    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43f5087-4ed9-4e95-bf72-57432ed7681c-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:20.469364 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.469330    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43f5087-4ed9-4e95-bf72-57432ed7681c-kube-api-access-v5crq" pod "e43f5087-4ed9-4e95-bf72-57432ed7681c" (UID: "e43f5087-4ed9-4e95-bf72-57432ed7681c"). InnerVolumeSpecName "kube-api-access-v5crq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:20.469364 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.469358    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6df122a-3baa-4fba-b3b0-4e6521e682d8-kube-api-access-6mwrj" pod "b6df122a-3baa-4fba-b3b0-4e6521e682d8" (UID: "b6df122a-3baa-4fba-b3b0-4e6521e682d8"). InnerVolumeSpecName "kube-api-access-6mwrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:20.569219 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.569181    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mwrj\" (UniqueName: \"kubernetes.io/projected/b6df122a-3baa-4fba-b3b0-4e6521e682d8-kube-api-access-6mwrj\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:20.569219 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.569215    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5crq\" (UniqueName: \"kubernetes.io/projected/e43f5087-4ed9-4e95-bf72-57432ed7681c-kube-api-access-v5crq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:20.893056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:20.893016    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g" podUID="e43f5087-4ed9-4e95-bf72-57432ed7681c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.79:50052\" within 1s: context deadline exceeded"
Nov 03 21:12:21.007496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.007461    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"
Nov 03 21:12:21.007496 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.007490    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g" event={"ID":"e43f5087-4ed9-4e95-bf72-57432ed7681c","Type":"ContainerDied","Data":"e90b3d97ff0c5d9a648a7e68c49988860913b2e42b0e491ff7b3e3a172194b95"}
Nov 03 21:12:21.007813 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.007546    5196 scope.go:122] "RemoveContainer" containerID="7ebb7666397c9416561479b2e12563ad152873ac8025dbee2777a3b0f9b5b383"
Nov 03 21:12:21.009530 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.009500    5196 generic.go:356] "Generic (PLEG): container finished" podID="b6df122a-3baa-4fba-b3b0-4e6521e682d8" containerID="359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a" exitCode=0
Nov 03 21:12:21.009664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.009545    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"
Nov 03 21:12:21.009664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.009586    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8" event={"ID":"b6df122a-3baa-4fba-b3b0-4e6521e682d8","Type":"ContainerDied","Data":"359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a"}
Nov 03 21:12:21.009664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.009643    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8" event={"ID":"b6df122a-3baa-4fba-b3b0-4e6521e682d8","Type":"ContainerDied","Data":"da5994d5b3621fb608e2a7331f8b5245231ea589d986bda1b3720932d06532ff"}
Nov 03 21:12:21.021305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.021278    5196 scope.go:122] "RemoveContainer" containerID="359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a"
Nov 03 21:12:21.033406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.033384    5196 scope.go:122] "RemoveContainer" containerID="359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a"
Nov 03 21:12:21.033746 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:12:21.033721    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a\": container with ID starting with 359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a not found: ID does not exist" containerID="359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a"
Nov 03 21:12:21.033809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.033758    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a"} err="failed to get container status \"359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a\": rpc error: code = NotFound desc = could not find container \"359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a\": container with ID starting with 359c5dd702a7a2ed3f580e2b7af8face456e68059ee00a5eb2f14a0cd63cf70a not found: ID does not exist"
Nov 03 21:12:21.162727 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.162638    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"]
Nov 03 21:12:21.210027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.209986    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-41aa21-6579dd577-ndr9g"]
Nov 03 21:12:21.312823 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.312783    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"]
Nov 03 21:12:21.316475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:21.316450    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-41aa21-7fc5c798d7-6ghk8"]
Nov 03 21:12:22.110432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:22.110392    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6df122a-3baa-4fba-b3b0-4e6521e682d8" path="/var/lib/kubelet/pods/b6df122a-3baa-4fba-b3b0-4e6521e682d8/volumes"
Nov 03 21:12:22.111015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:22.110994    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43f5087-4ed9-4e95-bf72-57432ed7681c" path="/var/lib/kubelet/pods/e43f5087-4ed9-4e95-bf72-57432ed7681c/volumes"
Nov 03 21:12:22.858400 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:22.858356    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" containerID="cri-o://f83a45737dcae38b5f46924525ec4286353435f642d27402b88b4391ba670887" gracePeriod=45
Nov 03 21:12:22.906751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:22.906714    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main" containerID="cri-o://bc4377b760a348c2c5870b4b4ff25c100554a68ad782a586cd82947b16b15774" gracePeriod=45
Nov 03 21:12:22.908915 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:22.908886    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" containerID="cri-o://3928a45ee9a2b53758cf56cc9e9b9a7ee53f5b3155986f5931c90f74957188f9" gracePeriod=45
Nov 03 21:12:23.033922 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.033863    5196 generic.go:356] "Generic (PLEG): container finished" podID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerID="3928a45ee9a2b53758cf56cc9e9b9a7ee53f5b3155986f5931c90f74957188f9" exitCode=0
Nov 03 21:12:23.034233 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.033957    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" event={"ID":"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b","Type":"ContainerDied","Data":"3928a45ee9a2b53758cf56cc9e9b9a7ee53f5b3155986f5931c90f74957188f9"}
Nov 03 21:12:23.036627 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.036595    5196 generic.go:356] "Generic (PLEG): container finished" podID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerID="f83a45737dcae38b5f46924525ec4286353435f642d27402b88b4391ba670887" exitCode=0
Nov 03 21:12:23.036827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.036682    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" event={"ID":"784d401d-fb0d-4d5f-8bfb-01c80bc6c179","Type":"ContainerDied","Data":"f83a45737dcae38b5f46924525ec4286353435f642d27402b88b4391ba670887"}
Nov 03 21:12:23.040254 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.040222    5196 generic.go:356] "Generic (PLEG): container finished" podID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerID="bc4377b760a348c2c5870b4b4ff25c100554a68ad782a586cd82947b16b15774" exitCode=0
Nov 03 21:12:23.040483 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.040426    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" event={"ID":"0fe9d04c-600c-4be8-82b4-a2a80b23f05d","Type":"ContainerDied","Data":"bc4377b760a348c2c5870b4b4ff25c100554a68ad782a586cd82947b16b15774"}
Nov 03 21:12:23.199259 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.199226    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 21:12:23.200128 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200080    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-model-cache\") pod \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") "
Nov 03 21:12:23.200275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200165    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tmp-dir\") pod \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") "
Nov 03 21:12:23.200341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200283    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-kube-api-access-l6vrt\" (UniqueName: \"kubernetes.io/projected/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-kube-api-access-l6vrt\") pod \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") "
Nov 03 21:12:23.200341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200326    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tls-certs\") pod \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") "
Nov 03 21:12:23.200459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200413    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-model-cache" pod "0fe9d04c-600c-4be8-82b4-a2a80b23f05d" (UID: "0fe9d04c-600c-4be8-82b4-a2a80b23f05d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.200459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200447    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-home\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-home\") pod \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") "
Nov 03 21:12:23.200979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200705    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-home" pod "0fe9d04c-600c-4be8-82b4-a2a80b23f05d" (UID: "0fe9d04c-600c-4be8-82b4-a2a80b23f05d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.200979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200810    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-dshm\") pod \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\" (UID: \"0fe9d04c-600c-4be8-82b4-a2a80b23f05d\") "
Nov 03 21:12:23.200979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.200945    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tmp-dir" pod "0fe9d04c-600c-4be8-82b4-a2a80b23f05d" (UID: "0fe9d04c-600c-4be8-82b4-a2a80b23f05d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.201399 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.201376    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.201533 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.201402    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.201533 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.201416    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.206048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.206022    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-dshm" pod "0fe9d04c-600c-4be8-82b4-a2a80b23f05d" (UID: "0fe9d04c-600c-4be8-82b4-a2a80b23f05d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.216531 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.216483    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tls-certs" pod "0fe9d04c-600c-4be8-82b4-a2a80b23f05d" (UID: "0fe9d04c-600c-4be8-82b4-a2a80b23f05d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:23.218577 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.218549    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-kube-api-access-l6vrt" pod "0fe9d04c-600c-4be8-82b4-a2a80b23f05d" (UID: "0fe9d04c-600c-4be8-82b4-a2a80b23f05d"). InnerVolumeSpecName "kube-api-access-l6vrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:23.235420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.235382    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 21:12:23.246076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.246046    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 21:12:23.302762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.302717    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tls-certs\" (UniqueName: \"kubernetes.io/secret/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tls-certs\") pod \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") "
Nov 03 21:12:23.302937 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.302793    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-kube-api-access-mr6z2\" (UniqueName: \"kubernetes.io/projected/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-kube-api-access-mr6z2\") pod \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") "
Nov 03 21:12:23.302937 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.302820    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tmp-dir\") pod \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") "
Nov 03 21:12:23.302937 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.302919    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-home\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-home\") pod \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") "
Nov 03 21:12:23.303162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.302953    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tls-certs\") pod \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") "
Nov 03 21:12:23.303162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.302984    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tmp-dir\") pod \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") "
Nov 03 21:12:23.303162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303018    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-home\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-home\") pod \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") "
Nov 03 21:12:23.303162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303045    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-model-cache\") pod \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") "
Nov 03 21:12:23.303162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303147    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-dshm\") pod \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") "
Nov 03 21:12:23.303441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303181    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-kube-api-access-dgtpp\" (UniqueName: \"kubernetes.io/projected/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-kube-api-access-dgtpp\") pod \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") "
Nov 03 21:12:23.303441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303229    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-dshm\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-dshm\") pod \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\" (UID: \"784d401d-fb0d-4d5f-8bfb-01c80bc6c179\") "
Nov 03 21:12:23.303441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303324    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-model-cache\") pod \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\" (UID: \"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b\") "
Nov 03 21:12:23.303603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303223    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tmp-dir" pod "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" (UID: "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.303603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303236    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-home" pod "784d401d-fb0d-4d5f-8bfb-01c80bc6c179" (UID: "784d401d-fb0d-4d5f-8bfb-01c80bc6c179"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.303603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303451    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tmp-dir" pod "784d401d-fb0d-4d5f-8bfb-01c80bc6c179" (UID: "784d401d-fb0d-4d5f-8bfb-01c80bc6c179"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.303603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303400    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-model-cache" pod "784d401d-fb0d-4d5f-8bfb-01c80bc6c179" (UID: "784d401d-fb0d-4d5f-8bfb-01c80bc6c179"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.303603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303435    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-home" pod "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" (UID: "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.303867 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303766    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-model-cache" pod "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" (UID: "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.304000 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.303978    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.304121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.304006    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.304121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.304021    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.304121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.304037    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vrt\" (UniqueName: \"kubernetes.io/projected/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-kube-api-access-l6vrt\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.304121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.304052    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.304527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.304265    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.304527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.304280    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.304527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.304293    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe9d04c-600c-4be8-82b4-a2a80b23f05d-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.304527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.304305    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.306056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.306030    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tls-certs" pod "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" (UID: "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:23.306239 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.306064    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-dshm" pod "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" (UID: "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.306466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.306320    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tls-certs" pod "784d401d-fb0d-4d5f-8bfb-01c80bc6c179" (UID: "784d401d-fb0d-4d5f-8bfb-01c80bc6c179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:23.306466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.306361    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-kube-api-access-dgtpp" pod "784d401d-fb0d-4d5f-8bfb-01c80bc6c179" (UID: "784d401d-fb0d-4d5f-8bfb-01c80bc6c179"). InnerVolumeSpecName "kube-api-access-dgtpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:23.306616 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.306486    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-dshm" pod "784d401d-fb0d-4d5f-8bfb-01c80bc6c179" (UID: "784d401d-fb0d-4d5f-8bfb-01c80bc6c179"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:12:23.306794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.306770    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-kube-api-access-mr6z2" pod "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" (UID: "1f922216-cc27-48ab-99c2-e2dd7fc9ba3b"). InnerVolumeSpecName "kube-api-access-mr6z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:23.405165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.405037    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.405165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.405069    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr6z2\" (UniqueName: \"kubernetes.io/projected/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-kube-api-access-mr6z2\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.405165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.405079    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.405165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.405087    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.405165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.405134    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtpp\" (UniqueName: \"kubernetes.io/projected/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-kube-api-access-dgtpp\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.405165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.405146    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784d401d-fb0d-4d5f-8bfb-01c80bc6c179-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:23.569905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:23.569868    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:24.050265 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.050234    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"
Nov 03 21:12:24.050602 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.050260    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" event={"ID":"784d401d-fb0d-4d5f-8bfb-01c80bc6c179","Type":"ContainerDied","Data":"6ac82ed337be18d402f1232309160b856c7a0a920dafb54cd33c23a210afc1fe"}
Nov 03 21:12:24.050602 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.050325    5196 scope.go:122] "RemoveContainer" containerID="f83a45737dcae38b5f46924525ec4286353435f642d27402b88b4391ba670887"
Nov 03 21:12:24.051808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.051787    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"
Nov 03 21:12:24.051870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.051804    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7" event={"ID":"0fe9d04c-600c-4be8-82b4-a2a80b23f05d","Type":"ContainerDied","Data":"4ba5d7d862bcaa19720da9e2bd7a92d7647a980efcfb2d3002536ce157ae57d6"}
Nov 03 21:12:24.058384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.058355    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" event={"ID":"1f922216-cc27-48ab-99c2-e2dd7fc9ba3b","Type":"ContainerDied","Data":"050ce90bed1406bfbdb8864a10fe696fb4ad275216c785d5052cbd99b0daf107"}
Nov 03 21:12:24.058504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.058404    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"
Nov 03 21:12:24.062463 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.062440    5196 scope.go:122] "RemoveContainer" containerID="bc4377b760a348c2c5870b4b4ff25c100554a68ad782a586cd82947b16b15774"
Nov 03 21:12:24.074347 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.074325    5196 scope.go:122] "RemoveContainer" containerID="3928a45ee9a2b53758cf56cc9e9b9a7ee53f5b3155986f5931c90f74957188f9"
Nov 03 21:12:24.138825 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.138727    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:12:24.177964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.177910    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:12:24.205636 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.205601    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"]
Nov 03 21:12:24.210206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.210169    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/simulator-w1-41aa21-kserve-5f6c4888d-l5kvs"]
Nov 03 21:12:24.311417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.311371    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"]
Nov 03 21:12:24.315679 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.315654    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-2-w1-41aa21-kserve-d8d98dc9-jtkhz"]
Nov 03 21:12:24.417532 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.417438    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"]
Nov 03 21:12:24.422551 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:24.422521    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-41aa21/distinct-w1-41aa21-kserve-76d4997fcb-7crd7"]
Nov 03 21:12:26.110972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:26.110936    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" path="/var/lib/kubelet/pods/0fe9d04c-600c-4be8-82b4-a2a80b23f05d/volumes"
Nov 03 21:12:26.111481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:26.111421    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" path="/var/lib/kubelet/pods/1f922216-cc27-48ab-99c2-e2dd7fc9ba3b/volumes"
Nov 03 21:12:26.111806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:26.111792    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" path="/var/lib/kubelet/pods/784d401d-fb0d-4d5f-8bfb-01c80bc6c179/volumes"
Nov 03 21:12:26.624114 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:26.617867    5196 scope.go:122] "RemoveContainer" containerID="3e28c93829dfe8c69ea66225bb5823cc657586add841eab99cc452e4abf68cca"
Nov 03 21:12:26.691117 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:26.688042    5196 scope.go:122] "RemoveContainer" containerID="bed3abef20e5581cfe01c177ef74ebc89e661debbc0c39c2f36990a536bb9c52"
Nov 03 21:12:26.740563 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:26.740536    5196 scope.go:122] "RemoveContainer" containerID="f573ecb6e16a453e53ebd46b6eb5e561961c56012624b483eebc3450714a3b26"
Nov 03 21:12:26.796513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:26.796486    5196 scope.go:122] "RemoveContainer" containerID="769eff51ef1acbc17fbc6eb6f6fdcd24ae79cacfc0ad940168dc116d798b94c4"
Nov 03 21:12:28.060120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:28.059221    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:12:28.060120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:28.059320    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:12:28.627582 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:28.627544    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:29.045555 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:29.045472    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"]
Nov 03 21:12:29.092257 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:29.092221    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869da0d9-lksww"]
Nov 03 21:12:30.109352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:30.109321    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cc87a0-5cf0-4b7c-a72b-75fa1435b92b" path="/var/lib/kubelet/pods/51cc87a0-5cf0-4b7c-a72b-75fa1435b92b/volumes"
Nov 03 21:12:33.349009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:33.348975    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:38.162846 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:38.162806    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:42.867905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:42.867860    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:44.060466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:44.060428    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:45.139176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:45.139135    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:45.334392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:45.334351    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:45.478762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:45.478667    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:47.104753 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:47.104720    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:12:47.576732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:47.576695    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:48.486754 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:12:48.486712    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:12:50.471330 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:50.471283    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:50.978946 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:50.978906    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:51.181978 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.181932    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:51.789251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.789222    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"]
Nov 03 21:12:51.790112 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790073    5196 cpu_manager.go:414] "containerMap: removing container" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main"
Nov 03 21:12:51.790112 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790112    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790133    5196 cpu_manager.go:414] "containerMap: removing container" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790140    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790151    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e446063e-9950-4ca5-83fc-ef2920a151d2" containerName="revoke-keys"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790158    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e446063e-9950-4ca5-83fc-ef2920a151d2" containerName="revoke-keys"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790171    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e43f5087-4ed9-4e95-bf72-57432ed7681c" containerName="payload-processing"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790179    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43f5087-4ed9-4e95-bf72-57432ed7681c" containerName="payload-processing"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790187    5196 cpu_manager.go:414] "containerMap: removing container" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790194    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790218    5196 cpu_manager.go:414] "containerMap: removing container" podUID="b6df122a-3baa-4fba-b3b0-4e6521e682d8" containerName="payload-pre-processing"
Nov 03 21:12:51.790334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790225    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6df122a-3baa-4fba-b3b0-4e6521e682d8" containerName="payload-pre-processing"
Nov 03 21:12:51.790844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790368    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe9d04c-600c-4be8-82b4-a2a80b23f05d" containerName="main"
Nov 03 21:12:51.790844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790386    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="784d401d-fb0d-4d5f-8bfb-01c80bc6c179" containerName="main"
Nov 03 21:12:51.790844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790398    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e446063e-9950-4ca5-83fc-ef2920a151d2" containerName="revoke-keys"
Nov 03 21:12:51.790844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790409    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43f5087-4ed9-4e95-bf72-57432ed7681c" containerName="payload-processing"
Nov 03 21:12:51.790844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790420    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f922216-cc27-48ab-99c2-e2dd7fc9ba3b" containerName="main"
Nov 03 21:12:51.790844 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.790433    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6df122a-3baa-4fba-b3b0-4e6521e682d8" containerName="payload-pre-processing"
Nov 03 21:12:51.794384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.794364    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:51.844171 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.844124    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"]
Nov 03 21:12:51.849543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.849515    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"]
Nov 03 21:12:51.849730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.849625    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:51.887318 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.887285    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:12:51.887515 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.887373    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:12:51.901465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.901417    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"]
Nov 03 21:12:51.902194 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.902168    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:51.902354 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.902334    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:51.902417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:51.902368    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vchq7\" (UniqueName: \"kubernetes.io/projected/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-kube-api-access-vchq7\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:52.003797 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.003757    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:52.004026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.003813    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63935a9f-8280-4535-b770-c77191bf297b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.004026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.003845    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2fl\" (UniqueName: \"kubernetes.io/projected/63935a9f-8280-4535-b770-c77191bf297b-kube-api-access-8h2fl\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.004026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.003875    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63935a9f-8280-4535-b770-c77191bf297b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.004026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.003903    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:52.004328 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.004056    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vchq7\" (UniqueName: \"kubernetes.io/projected/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-kube-api-access-vchq7\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:52.004648 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.004628    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:52.006493 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.006473    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:52.057782 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.057698    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vchq7\" (UniqueName: \"kubernetes.io/projected/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-kube-api-access-vchq7\") pod \"maas-api-revoke-sub-b2c7089a-4tgcl\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:52.105521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.105479    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63935a9f-8280-4535-b770-c77191bf297b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.105742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.105534    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2fl\" (UniqueName: \"kubernetes.io/projected/63935a9f-8280-4535-b770-c77191bf297b-kube-api-access-8h2fl\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.105742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.105577    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63935a9f-8280-4535-b770-c77191bf297b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.106376 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.106347    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63935a9f-8280-4535-b770-c77191bf297b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.108559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.108534    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:52.108959 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.108937    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63935a9f-8280-4535-b770-c77191bf297b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.159527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.159498    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2fl\" (UniqueName: \"kubernetes.io/projected/63935a9f-8280-4535-b770-c77191bf297b-kube-api-access-8h2fl\") pod \"maas-api-revoke-sub-596e08cc-pgrcd\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.167255 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.166508    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:52.486950 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.486915    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"]
Nov 03 21:12:52.493305 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:12:52.493274    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bf1caa_27f5_4366_89b5_cc88e17b58fa.slice/crio-f3269152a99f0094553deac48756f409078bcb91dc8d282893d6cef1c3d101be WatchSource:0}: Error finding container f3269152a99f0094553deac48756f409078bcb91dc8d282893d6cef1c3d101be: Status 404 returned error can't find the container with id f3269152a99f0094553deac48756f409078bcb91dc8d282893d6cef1c3d101be
Nov 03 21:12:52.594661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.594624    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"]
Nov 03 21:12:52.601029 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:12:52.601002    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63935a9f_8280_4535_b770_c77191bf297b.slice/crio-c1dcafeaa9bc0420d5bd287c654b49ee211e2e712cea3c0f27fe57248138635b WatchSource:0}: Error finding container c1dcafeaa9bc0420d5bd287c654b49ee211e2e712cea3c0f27fe57248138635b: Status 404 returned error can't find the container with id c1dcafeaa9bc0420d5bd287c654b49ee211e2e712cea3c0f27fe57248138635b
Nov 03 21:12:52.666455 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:52.666421    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:53.347985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:53.347948    5196 generic.go:356] "Generic (PLEG): container finished" podID="e7bf1caa-27f5-4366-89b5-cc88e17b58fa" containerID="eee804c0c91444c6b166db4240cd1596878b1139b0c67c707017feaf2772c609" exitCode=0
Nov 03 21:12:53.348440 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:53.348024    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl" event={"ID":"e7bf1caa-27f5-4366-89b5-cc88e17b58fa","Type":"ContainerDied","Data":"eee804c0c91444c6b166db4240cd1596878b1139b0c67c707017feaf2772c609"}
Nov 03 21:12:53.348440 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:53.348081    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl" event={"ID":"e7bf1caa-27f5-4366-89b5-cc88e17b58fa","Type":"ContainerStarted","Data":"f3269152a99f0094553deac48756f409078bcb91dc8d282893d6cef1c3d101be"}
Nov 03 21:12:53.349990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:53.349958    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd" event={"ID":"63935a9f-8280-4535-b770-c77191bf297b","Type":"ContainerStarted","Data":"81e69aeae209f5f16b65a7411303daf854b980c65a09d2441d29cc879d6b5254"}
Nov 03 21:12:53.350190 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:53.350009    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd" event={"ID":"63935a9f-8280-4535-b770-c77191bf297b","Type":"ContainerStarted","Data":"c1dcafeaa9bc0420d5bd287c654b49ee211e2e712cea3c0f27fe57248138635b"}
Nov 03 21:12:53.562470 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:53.562404    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd" podStartSLOduration=2.562386134 podStartE2EDuration="2.562386134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:53.560298755 +0000 UTC m=+2613.564257498" watchObservedRunningTime="2026-11-03 21:12:53.562386134 +0000 UTC m=+2613.566344917"
Nov 03 21:12:54.360386 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.360307    5196 generic.go:356] "Generic (PLEG): container finished" podID="63935a9f-8280-4535-b770-c77191bf297b" containerID="81e69aeae209f5f16b65a7411303daf854b980c65a09d2441d29cc879d6b5254" exitCode=0
Nov 03 21:12:54.360822 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.360399    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd" event={"ID":"63935a9f-8280-4535-b770-c77191bf297b","Type":"ContainerDied","Data":"81e69aeae209f5f16b65a7411303daf854b980c65a09d2441d29cc879d6b5254"}
Nov 03 21:12:54.534482 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.534456    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:54.636013 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.635927    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-kube-api-access-vchq7\" (UniqueName: \"kubernetes.io/projected/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-kube-api-access-vchq7\") pod \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") "
Nov 03 21:12:54.636013 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.635970    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-xks-ca\") pod \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") "
Nov 03 21:12:54.636013 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.636014    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-openshift-ca\") pod \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\" (UID: \"e7bf1caa-27f5-4366-89b5-cc88e17b58fa\") "
Nov 03 21:12:54.636478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.636448    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-openshift-ca" pod "e7bf1caa-27f5-4366-89b5-cc88e17b58fa" (UID: "e7bf1caa-27f5-4366-89b5-cc88e17b58fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:54.636617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.636575    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:54.638385 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.638362    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-kube-api-access-vchq7" pod "e7bf1caa-27f5-4366-89b5-cc88e17b58fa" (UID: "e7bf1caa-27f5-4366-89b5-cc88e17b58fa"). InnerVolumeSpecName "kube-api-access-vchq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:54.638504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.638406    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-xks-ca" pod "e7bf1caa-27f5-4366-89b5-cc88e17b58fa" (UID: "e7bf1caa-27f5-4366-89b5-cc88e17b58fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:54.738402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.738364    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vchq7\" (UniqueName: \"kubernetes.io/projected/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-kube-api-access-vchq7\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:54.738402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:54.738402    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bf1caa-27f5-4366-89b5-cc88e17b58fa-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:55.370405 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.370369    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"
Nov 03 21:12:55.370845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.370399    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl" event={"ID":"e7bf1caa-27f5-4366-89b5-cc88e17b58fa","Type":"ContainerDied","Data":"f3269152a99f0094553deac48756f409078bcb91dc8d282893d6cef1c3d101be"}
Nov 03 21:12:55.370845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.370450    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3269152a99f0094553deac48756f409078bcb91dc8d282893d6cef1c3d101be"
Nov 03 21:12:55.521908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.521876    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:55.649448 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.649367    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63935a9f-8280-4535-b770-c77191bf297b-kube-api-access-8h2fl\" (UniqueName: \"kubernetes.io/projected/63935a9f-8280-4535-b770-c77191bf297b-kube-api-access-8h2fl\") pod \"63935a9f-8280-4535-b770-c77191bf297b\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") "
Nov 03 21:12:55.649633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.649535    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63935a9f-8280-4535-b770-c77191bf297b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63935a9f-8280-4535-b770-c77191bf297b-maas-api-xks-ca\") pod \"63935a9f-8280-4535-b770-c77191bf297b\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") "
Nov 03 21:12:55.649703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.649675    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63935a9f-8280-4535-b770-c77191bf297b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63935a9f-8280-4535-b770-c77191bf297b-maas-api-openshift-ca\") pod \"63935a9f-8280-4535-b770-c77191bf297b\" (UID: \"63935a9f-8280-4535-b770-c77191bf297b\") "
Nov 03 21:12:55.650126 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.650071    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63935a9f-8280-4535-b770-c77191bf297b-maas-api-openshift-ca" pod "63935a9f-8280-4535-b770-c77191bf297b" (UID: "63935a9f-8280-4535-b770-c77191bf297b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:55.651783 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.651759    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63935a9f-8280-4535-b770-c77191bf297b-kube-api-access-8h2fl" pod "63935a9f-8280-4535-b770-c77191bf297b" (UID: "63935a9f-8280-4535-b770-c77191bf297b"). InnerVolumeSpecName "kube-api-access-8h2fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:55.652289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.652271    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63935a9f-8280-4535-b770-c77191bf297b-maas-api-xks-ca" pod "63935a9f-8280-4535-b770-c77191bf297b" (UID: "63935a9f-8280-4535-b770-c77191bf297b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:55.751763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.751715    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63935a9f-8280-4535-b770-c77191bf297b-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:55.751763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.751753    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63935a9f-8280-4535-b770-c77191bf297b-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:55.751763 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:55.751768    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h2fl\" (UniqueName: \"kubernetes.io/projected/63935a9f-8280-4535-b770-c77191bf297b-kube-api-access-8h2fl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:56.380079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:56.380004    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"
Nov 03 21:12:56.380079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:56.380033    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd" event={"ID":"63935a9f-8280-4535-b770-c77191bf297b","Type":"ContainerDied","Data":"c1dcafeaa9bc0420d5bd287c654b49ee211e2e712cea3c0f27fe57248138635b"}
Nov 03 21:12:56.380529 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:56.380081    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1dcafeaa9bc0420d5bd287c654b49ee211e2e712cea3c0f27fe57248138635b"
Nov 03 21:12:57.381134 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:12:57.381079    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:02.090206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:02.090171    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:06.872217 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:06.872173    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:11.675201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:11.675165    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:16.375121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:16.375069    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:18.485777 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:13:18.485742    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:13:20.870421 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:20.870375    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:13:21.074006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:21.073968    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:25.858853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:25.858810    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:27.334861 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:27.334830    5196 scope.go:122] "RemoveContainer" containerID="e4761a453badeba61606cfe414c760138c00324f9f5040b40dea7bb1345a3630"
Nov 03 21:13:30.561543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:30.561511    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:35.261840 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:35.261802    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:39.385229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:39.385191    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:40.944456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:40.944409    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:41.176620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:41.176588    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:41.318428 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:41.318390    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:41.569822 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:41.569729    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"]
Nov 03 21:13:41.616810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:41.616774    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-41aa21-pbzbx"]
Nov 03 21:13:41.915661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:41.915560    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:42.109247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:42.109205    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e446063e-9950-4ca5-83fc-ef2920a151d2" path="/var/lib/kubelet/pods/e446063e-9950-4ca5-83fc-ef2920a151d2/volumes"
Nov 03 21:13:46.271041 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:46.271000    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:47.981522 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:47.981484    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:48.133050 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:48.133014    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:48.275545 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:48.275458    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"]
Nov 03 21:13:48.275730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:48.275707    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" containerID="cri-o://0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a" gracePeriod=30
Nov 03 21:13:48.306674 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:48.306640    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:48.484910 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:13:48.484867    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:13:50.093970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:50.093922    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:50.244552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:50.244506    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:50.816283 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:50.816247    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:51.013809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:51.013754    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:52.922380 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:52.922343    5196 patch_prober.go:29] interesting pod/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:13:52.922805 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:52.922417    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:13:53.541010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.540985    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 21:13:53.559463 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559386    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-data\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.559463 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559442    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-certs\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.559463 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559463    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-podinfo\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.559748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559489    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-credential-socket\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.559748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559536    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5671dd3-4835-4f27-93db-f34fcca38ca4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5671dd3-4835-4f27-93db-f34fcca38ca4-istiod-ca-cert\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.559748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559571    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-envoy\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.559748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559668    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-token\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-token\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.559748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559697    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-kube-api-access-7k5m4\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-kube-api-access-7k5m4\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.559748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559726    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-data" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:53.560072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559745    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-certs" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:53.560072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559753    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-socket\") pod \"f5671dd3-4835-4f27-93db-f34fcca38ca4\" (UID: \"f5671dd3-4835-4f27-93db-f34fcca38ca4\") "
Nov 03 21:13:53.560072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559897    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-credential-socket" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:53.560072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.559964    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5671dd3-4835-4f27-93db-f34fcca38ca4-istiod-ca-cert" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:13:53.560609 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.560307    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-socket" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:53.560829 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.560802    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.560954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.560832    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.560954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.560858    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.560954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.560874    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5671dd3-4835-4f27-93db-f34fcca38ca4-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.562754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.562478    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-podinfo" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:13:53.562754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.562526    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-kube-api-access-7k5m4" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "kube-api-access-7k5m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:53.573190 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.573144    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-envoy" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:53.573609 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.573587    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-token" pod "f5671dd3-4835-4f27-93db-f34fcca38ca4" (UID: "f5671dd3-4835-4f27-93db-f34fcca38ca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:53.662253 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.662212    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.662253 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.662250    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.662253 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.662259    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.662516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.662269    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.662516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.662279    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k5m4\" (UniqueName: \"kubernetes.io/projected/f5671dd3-4835-4f27-93db-f34fcca38ca4-kube-api-access-7k5m4\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:53.859945 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.859904    5196 generic.go:356] "Generic (PLEG): container finished" podID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerID="0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a" exitCode=0
Nov 03 21:13:53.860155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.859976    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"
Nov 03 21:13:53.860155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.859976    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" event={"ID":"f5671dd3-4835-4f27-93db-f34fcca38ca4","Type":"ContainerDied","Data":"0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a"}
Nov 03 21:13:53.860155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.860083    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm" event={"ID":"f5671dd3-4835-4f27-93db-f34fcca38ca4","Type":"ContainerDied","Data":"e3a7dd2b5fb58153acdd47d5e6d437fbeb0df3bfe66c12e6c45af53e15709e4e"}
Nov 03 21:13:53.860155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.860129    5196 scope.go:122] "RemoveContainer" containerID="0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a"
Nov 03 21:13:53.873357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.873296    5196 scope.go:122] "RemoveContainer" containerID="0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a"
Nov 03 21:13:53.873641 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:13:53.873620    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a\": container with ID starting with 0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a not found: ID does not exist" containerID="0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a"
Nov 03 21:13:53.873704 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:53.873657    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a"} err="failed to get container status \"0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a\": rpc error: code = NotFound desc = could not find container \"0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a\": container with ID starting with 0027f7fb4951acc74527be2389ceb518477af10451305400d05f0389b232124a not found: ID does not exist"
Nov 03 21:13:54.018591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:54.018559    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"]
Nov 03 21:13:54.021853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:54.021827    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-41aa21-openshift-default-795986fdf5-hbvsm"]
Nov 03 21:13:54.108754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:54.108717    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" path="/var/lib/kubelet/pods/f5671dd3-4835-4f27-93db-f34fcca38ca4/volumes"
Nov 03 21:13:54.275640 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:54.275557    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:13:55.767232 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:13:55.767193    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:00.439783 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:00.439746    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:05.183936 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:05.183889    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:09.947299 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:09.947258    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:11.050889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:11.050854    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"]
Nov 03 21:14:11.098164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:11.098122    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"]
Nov 03 21:14:11.101632 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:11.101603    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14694eb2-g8cz2"]
Nov 03 21:14:11.104716 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:11.104678    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e8f4f22-k4ctz"]
Nov 03 21:14:12.109514 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:12.109468    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b00d510-0f2b-4896-8cf2-0bcb280bcdb8" path="/var/lib/kubelet/pods/4b00d510-0f2b-4896-8cf2-0bcb280bcdb8/volumes"
Nov 03 21:14:12.110148 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:12.110126    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef28e282-ce5f-479d-a403-7ea65ef54d24" path="/var/lib/kubelet/pods/ef28e282-ce5f-479d-a403-7ea65ef54d24/volumes"
Nov 03 21:14:14.668706 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:14.668663    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:18.485389 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:14:18.485354    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:14:18.807469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:18.807388    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:19.005769 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:19.005732    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:19.455935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:19.455896    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:20.639340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:20.639306    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:14:20.646043 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:20.646018    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:14:20.658368 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:20.658344    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:14:20.664250 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:20.664231    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:14:22.450367 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.450325    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"]
Nov 03 21:14:22.450995 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.450976    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy"
Nov 03 21:14:22.451070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.450999    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy"
Nov 03 21:14:22.451070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.451015    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e7bf1caa-27f5-4366-89b5-cc88e17b58fa" containerName="revoke-keys"
Nov 03 21:14:22.451070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.451020    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7bf1caa-27f5-4366-89b5-cc88e17b58fa" containerName="revoke-keys"
Nov 03 21:14:22.451070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.451034    5196 cpu_manager.go:414] "containerMap: removing container" podUID="63935a9f-8280-4535-b770-c77191bf297b" containerName="revoke-keys"
Nov 03 21:14:22.451070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.451039    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63935a9f-8280-4535-b770-c77191bf297b" containerName="revoke-keys"
Nov 03 21:14:22.451277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.451178    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5671dd3-4835-4f27-93db-f34fcca38ca4" containerName="istio-proxy"
Nov 03 21:14:22.451277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.451193    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7bf1caa-27f5-4366-89b5-cc88e17b58fa" containerName="revoke-keys"
Nov 03 21:14:22.451277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.451203    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="63935a9f-8280-4535-b770-c77191bf297b" containerName="revoke-keys"
Nov 03 21:14:22.454934 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.454912    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.507672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.507639    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"]
Nov 03 21:14:22.515839 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.515803    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.516018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.515940    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.516018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.515998    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfgm\" (UniqueName: \"kubernetes.io/projected/ebce0159-24c9-4a04-9125-6122296d96b3-kube-api-access-tcfgm\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.550031 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.549986    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:14:22.550250 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.550011    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:14:22.617357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.617320    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.617546 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.617373    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.617546 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.617402    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfgm\" (UniqueName: \"kubernetes.io/projected/ebce0159-24c9-4a04-9125-6122296d96b3-kube-api-access-tcfgm\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.618088 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.618060    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.619904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.619884    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.674174 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.674139    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfgm\" (UniqueName: \"kubernetes.io/projected/ebce0159-24c9-4a04-9125-6122296d96b3-kube-api-access-tcfgm\") pod \"maas-api-revoke-sub-00886018-xnh9l\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:22.769721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:22.769628    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:23.149811 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:23.149767    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"]
Nov 03 21:14:23.156008 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:14:23.155974    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebce0159_24c9_4a04_9125_6122296d96b3.slice/crio-7bf5922f06bb89fb55c7452f425e3fb4f30fc0ef48183e78f831d1543ebfda5f WatchSource:0}: Error finding container 7bf5922f06bb89fb55c7452f425e3fb4f30fc0ef48183e78f831d1543ebfda5f: Status 404 returned error can't find the container with id 7bf5922f06bb89fb55c7452f425e3fb4f30fc0ef48183e78f831d1543ebfda5f
Nov 03 21:14:24.096345 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.096247    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l" event={"ID":"ebce0159-24c9-4a04-9125-6122296d96b3","Type":"ContainerStarted","Data":"49c2e2f5ae94d4004e5a54da3e533775e23072efd9aff7a1e59cede9ddd23270"}
Nov 03 21:14:24.096345 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.096341    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l" event={"ID":"ebce0159-24c9-4a04-9125-6122296d96b3","Type":"ContainerStarted","Data":"7bf5922f06bb89fb55c7452f425e3fb4f30fc0ef48183e78f831d1543ebfda5f"}
Nov 03 21:14:24.112867 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.112832    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"]
Nov 03 21:14:24.116934 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.116913    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.135177 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.135136    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.135381 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.135353    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jv4\" (UniqueName: \"kubernetes.io/projected/501a5ad9-269c-4fac-be10-bfa97a1f52c8-kube-api-access-n4jv4\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.135471 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.135422    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.179694 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.179648    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:24.202276 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.202219    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"]
Nov 03 21:14:24.221240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.221173    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l" podStartSLOduration=2.221148042 podStartE2EDuration="2.221148042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:24.20288629 +0000 UTC m=+2704.206845043" watchObservedRunningTime="2026-11-03 21:14:24.221148042 +0000 UTC m=+2704.225106776"
Nov 03 21:14:24.237388 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.237352    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.237550 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.237462    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jv4\" (UniqueName: \"kubernetes.io/projected/501a5ad9-269c-4fac-be10-bfa97a1f52c8-kube-api-access-n4jv4\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.237550 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.237509    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.238831 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.238618    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.240334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.240294    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.291318 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.291284    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jv4\" (UniqueName: \"kubernetes.io/projected/501a5ad9-269c-4fac-be10-bfa97a1f52c8-kube-api-access-n4jv4\") pod \"maas-api-revoke-sub-52da04e1-zmjtz\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.431987 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.431894    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:24.812886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:24.812795    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"]
Nov 03 21:14:24.819233 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:14:24.819196    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501a5ad9_269c_4fac_be10_bfa97a1f52c8.slice/crio-ce791db28c383c8d0e30d4975c55d7d7b06a2e1b27270f136313638884acacc8 WatchSource:0}: Error finding container ce791db28c383c8d0e30d4975c55d7d7b06a2e1b27270f136313638884acacc8: Status 404 returned error can't find the container with id ce791db28c383c8d0e30d4975c55d7d7b06a2e1b27270f136313638884acacc8
Nov 03 21:14:25.105601 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:25.105561    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz" event={"ID":"501a5ad9-269c-4fac-be10-bfa97a1f52c8","Type":"ContainerStarted","Data":"ce791db28c383c8d0e30d4975c55d7d7b06a2e1b27270f136313638884acacc8"}
Nov 03 21:14:25.107378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:25.107345    5196 generic.go:356] "Generic (PLEG): container finished" podID="ebce0159-24c9-4a04-9125-6122296d96b3" containerID="49c2e2f5ae94d4004e5a54da3e533775e23072efd9aff7a1e59cede9ddd23270" exitCode=0
Nov 03 21:14:25.107502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:25.107389    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l" event={"ID":"ebce0159-24c9-4a04-9125-6122296d96b3","Type":"ContainerDied","Data":"49c2e2f5ae94d4004e5a54da3e533775e23072efd9aff7a1e59cede9ddd23270"}
Nov 03 21:14:26.115952 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.115919    5196 generic.go:356] "Generic (PLEG): container finished" podID="501a5ad9-269c-4fac-be10-bfa97a1f52c8" containerID="c948538ca077cf533c430898f5e88e64f9524dec1d017f89240c3b45c1155c6b" exitCode=0
Nov 03 21:14:26.116421 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.116004    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz" event={"ID":"501a5ad9-269c-4fac-be10-bfa97a1f52c8","Type":"ContainerDied","Data":"c948538ca077cf533c430898f5e88e64f9524dec1d017f89240c3b45c1155c6b"}
Nov 03 21:14:26.270647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.270622    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:26.362935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.362897    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-xks-ca\") pod \"ebce0159-24c9-4a04-9125-6122296d96b3\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") "
Nov 03 21:14:26.362935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.362936    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-openshift-ca\") pod \"ebce0159-24c9-4a04-9125-6122296d96b3\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") "
Nov 03 21:14:26.363263 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.362999    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebce0159-24c9-4a04-9125-6122296d96b3-kube-api-access-tcfgm\" (UniqueName: \"kubernetes.io/projected/ebce0159-24c9-4a04-9125-6122296d96b3-kube-api-access-tcfgm\") pod \"ebce0159-24c9-4a04-9125-6122296d96b3\" (UID: \"ebce0159-24c9-4a04-9125-6122296d96b3\") "
Nov 03 21:14:26.363460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.363420    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-openshift-ca" pod "ebce0159-24c9-4a04-9125-6122296d96b3" (UID: "ebce0159-24c9-4a04-9125-6122296d96b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:26.363638 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.363619    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:26.365417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.365389    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-xks-ca" pod "ebce0159-24c9-4a04-9125-6122296d96b3" (UID: "ebce0159-24c9-4a04-9125-6122296d96b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:14:26.365801 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.365781    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebce0159-24c9-4a04-9125-6122296d96b3-kube-api-access-tcfgm" pod "ebce0159-24c9-4a04-9125-6122296d96b3" (UID: "ebce0159-24c9-4a04-9125-6122296d96b3"). InnerVolumeSpecName "kube-api-access-tcfgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:26.465902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.465864    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebce0159-24c9-4a04-9125-6122296d96b3-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:26.465902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:26.465901    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcfgm\" (UniqueName: \"kubernetes.io/projected/ebce0159-24c9-4a04-9125-6122296d96b3-kube-api-access-tcfgm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:27.124661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.124629    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"
Nov 03 21:14:27.125188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.124683    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l" event={"ID":"ebce0159-24c9-4a04-9125-6122296d96b3","Type":"ContainerDied","Data":"7bf5922f06bb89fb55c7452f425e3fb4f30fc0ef48183e78f831d1543ebfda5f"}
Nov 03 21:14:27.125188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.124760    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bf5922f06bb89fb55c7452f425e3fb4f30fc0ef48183e78f831d1543ebfda5f"
Nov 03 21:14:27.289884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.289858    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:27.362548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.362505    5196 scope.go:122] "RemoveContainer" containerID="5c7f4d70bb246b7816229acc245065d5b25c5aa30f7051f64e266b2142fd8140"
Nov 03 21:14:27.376794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.376738    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501a5ad9-269c-4fac-be10-bfa97a1f52c8-kube-api-access-n4jv4\" (UniqueName: \"kubernetes.io/projected/501a5ad9-269c-4fac-be10-bfa97a1f52c8-kube-api-access-n4jv4\") pod \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") "
Nov 03 21:14:27.376964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.376945    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-xks-ca\") pod \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") "
Nov 03 21:14:27.377039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.376985    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-openshift-ca\") pod \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\" (UID: \"501a5ad9-269c-4fac-be10-bfa97a1f52c8\") "
Nov 03 21:14:27.377883 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.377854    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-openshift-ca" pod "501a5ad9-269c-4fac-be10-bfa97a1f52c8" (UID: "501a5ad9-269c-4fac-be10-bfa97a1f52c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:27.379390 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.379363    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-xks-ca" pod "501a5ad9-269c-4fac-be10-bfa97a1f52c8" (UID: "501a5ad9-269c-4fac-be10-bfa97a1f52c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:14:27.379610 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.379446    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501a5ad9-269c-4fac-be10-bfa97a1f52c8-kube-api-access-n4jv4" pod "501a5ad9-269c-4fac-be10-bfa97a1f52c8" (UID: "501a5ad9-269c-4fac-be10-bfa97a1f52c8"). InnerVolumeSpecName "kube-api-access-n4jv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:27.478701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.478657    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jv4\" (UniqueName: \"kubernetes.io/projected/501a5ad9-269c-4fac-be10-bfa97a1f52c8-kube-api-access-n4jv4\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:27.478701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.478694    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:27.478701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:27.478705    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a5ad9-269c-4fac-be10-bfa97a1f52c8-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:28.134134 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:28.134080    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz" event={"ID":"501a5ad9-269c-4fac-be10-bfa97a1f52c8","Type":"ContainerDied","Data":"ce791db28c383c8d0e30d4975c55d7d7b06a2e1b27270f136313638884acacc8"}
Nov 03 21:14:28.134134 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:28.134127    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"
Nov 03 21:14:28.134717 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:28.134148    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce791db28c383c8d0e30d4975c55d7d7b06a2e1b27270f136313638884acacc8"
Nov 03 21:14:28.873804 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:28.873765    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:33.672083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:33.672043    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:34.410482 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:34.410419    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:35.397766 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.397724    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"]
Nov 03 21:14:35.398360 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.398346    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ebce0159-24c9-4a04-9125-6122296d96b3" containerName="revoke-keys"
Nov 03 21:14:35.398416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.398362    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebce0159-24c9-4a04-9125-6122296d96b3" containerName="revoke-keys"
Nov 03 21:14:35.398416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.398399    5196 cpu_manager.go:414] "containerMap: removing container" podUID="501a5ad9-269c-4fac-be10-bfa97a1f52c8" containerName="revoke-keys"
Nov 03 21:14:35.398416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.398404    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501a5ad9-269c-4fac-be10-bfa97a1f52c8" containerName="revoke-keys"
Nov 03 21:14:35.398520 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.398510    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebce0159-24c9-4a04-9125-6122296d96b3" containerName="revoke-keys"
Nov 03 21:14:35.398563 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.398529    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="501a5ad9-269c-4fac-be10-bfa97a1f52c8" containerName="revoke-keys"
Nov 03 21:14:35.403761 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.403739    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.453776 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.453743    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"]
Nov 03 21:14:35.475147 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.475085    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxrf\" (UniqueName: \"kubernetes.io/projected/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-kube-api-access-ltxrf\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.475147 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.475152    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.475436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.475314    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.496325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.496287    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:14:35.496523 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.496389    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:14:35.576728 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.576679    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.576958 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.576864    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxrf\" (UniqueName: \"kubernetes.io/projected/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-kube-api-access-ltxrf\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.576958 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.576906    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.577597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.577570    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.579453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.579434    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.630934 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.630895    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxrf\" (UniqueName: \"kubernetes.io/projected/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-kube-api-access-ltxrf\") pod \"maas-api-revoke-sub-f8e998d6-99jqg\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:35.717960 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:35.717869    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:36.108462 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:14:36.108431    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fea920_3c6c_45dc_8f96_cb4c57ff175e.slice/crio-1badc9569184ab2f0c5ea6cef6403e6b7d15f5b12405bab58608409f87419346 WatchSource:0}: Error finding container 1badc9569184ab2f0c5ea6cef6403e6b7d15f5b12405bab58608409f87419346: Status 404 returned error can't find the container with id 1badc9569184ab2f0c5ea6cef6403e6b7d15f5b12405bab58608409f87419346
Nov 03 21:14:36.111203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:36.111130    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"]
Nov 03 21:14:36.198313 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:36.198277    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg" event={"ID":"e7fea920-3c6c-45dc-8f96-cb4c57ff175e","Type":"ContainerStarted","Data":"1badc9569184ab2f0c5ea6cef6403e6b7d15f5b12405bab58608409f87419346"}
Nov 03 21:14:37.207692 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:37.207652    5196 generic.go:356] "Generic (PLEG): container finished" podID="e7fea920-3c6c-45dc-8f96-cb4c57ff175e" containerID="5432978341b76514d9790a840613dcbb4193fb3c195c8ce5826dc2f66f595250" exitCode=0
Nov 03 21:14:37.208149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:37.207738    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg" event={"ID":"e7fea920-3c6c-45dc-8f96-cb4c57ff175e","Type":"ContainerDied","Data":"5432978341b76514d9790a840613dcbb4193fb3c195c8ce5826dc2f66f595250"}
Nov 03 21:14:38.388177 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.388144    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:38.474055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.473963    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:38.515595 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.515545    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-kube-api-access-ltxrf\" (UniqueName: \"kubernetes.io/projected/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-kube-api-access-ltxrf\") pod \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") "
Nov 03 21:14:38.516465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.515604    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-xks-ca\") pod \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") "
Nov 03 21:14:38.516465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.515723    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-openshift-ca\") pod \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\" (UID: \"e7fea920-3c6c-45dc-8f96-cb4c57ff175e\") "
Nov 03 21:14:38.516465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.516414    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-openshift-ca" pod "e7fea920-3c6c-45dc-8f96-cb4c57ff175e" (UID: "e7fea920-3c6c-45dc-8f96-cb4c57ff175e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:38.518931 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.518895    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-kube-api-access-ltxrf" pod "e7fea920-3c6c-45dc-8f96-cb4c57ff175e" (UID: "e7fea920-3c6c-45dc-8f96-cb4c57ff175e"). InnerVolumeSpecName "kube-api-access-ltxrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:38.519261 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.519237    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-xks-ca" pod "e7fea920-3c6c-45dc-8f96-cb4c57ff175e" (UID: "e7fea920-3c6c-45dc-8f96-cb4c57ff175e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:14:38.616953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.616916    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltxrf\" (UniqueName: \"kubernetes.io/projected/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-kube-api-access-ltxrf\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:38.616953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.616948    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:38.616953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:38.616958    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7fea920-3c6c-45dc-8f96-cb4c57ff175e-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:39.225193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:39.225147    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg" event={"ID":"e7fea920-3c6c-45dc-8f96-cb4c57ff175e","Type":"ContainerDied","Data":"1badc9569184ab2f0c5ea6cef6403e6b7d15f5b12405bab58608409f87419346"}
Nov 03 21:14:39.225401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:39.225212    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1badc9569184ab2f0c5ea6cef6403e6b7d15f5b12405bab58608409f87419346"
Nov 03 21:14:39.225401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:39.225161    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"
Nov 03 21:14:42.047172 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:42.047135    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"]
Nov 03 21:14:42.093952 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:42.093913    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c104393c-9rzng"]
Nov 03 21:14:42.097696 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:42.097661    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"]
Nov 03 21:14:42.101123 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:42.101072    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d882e4-mw64t"]
Nov 03 21:14:42.110119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:42.110065    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee45821b-f2a2-41e1-8250-bd8c74a7f7e3" path="/var/lib/kubelet/pods/ee45821b-f2a2-41e1-8250-bd8c74a7f7e3/volumes"
Nov 03 21:14:42.110593 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:42.110578    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcc8f5d-09cd-4a04-8213-dee7da6d120c" path="/var/lib/kubelet/pods/ffcc8f5d-09cd-4a04-8213-dee7da6d120c/volumes"
Nov 03 21:14:43.178481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:43.178447    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:47.200293 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:47.200254    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:14:48.034658 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:48.034617    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:48.485962 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:14:48.485921    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:14:51.183707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:51.183670    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:51.362779 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:51.362742    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:51.534370 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:51.534268    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"]
Nov 03 21:14:51.554042 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:51.553993    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:52.522194 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.522156    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"]
Nov 03 21:14:52.522781 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.522767    5196 cpu_manager.go:414] "containerMap: removing container" podUID="e7fea920-3c6c-45dc-8f96-cb4c57ff175e" containerName="revoke-keys"
Nov 03 21:14:52.522830 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.522782    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7fea920-3c6c-45dc-8f96-cb4c57ff175e" containerName="revoke-keys"
Nov 03 21:14:52.522878 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.522869    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7fea920-3c6c-45dc-8f96-cb4c57ff175e" containerName="revoke-keys"
Nov 03 21:14:52.527564 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.527545    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.576504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.576472    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:14:52.578636 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.578610    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"]
Nov 03 21:14:52.598189 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.598146    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.598382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.598214    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.598382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.598374    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qms\" (UniqueName: \"kubernetes.io/projected/1b0dad2f-a03a-452b-8fba-4a20b2905abf-kube-api-access-k7qms\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.621784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.621756    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:14:52.672660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.672615    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:52.700340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.699940    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qms\" (UniqueName: \"kubernetes.io/projected/1b0dad2f-a03a-452b-8fba-4a20b2905abf-kube-api-access-k7qms\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.700340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.700060    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.700340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.700126    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.703194 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.703160    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.707869 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.707800    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.757131 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.757083    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qms\" (UniqueName: \"kubernetes.io/projected/1b0dad2f-a03a-452b-8fba-4a20b2905abf-kube-api-access-k7qms\") pod \"maas-api-revoke-sub-6570bedb-mcd9j\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:52.842395 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:52.842354    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:53.246780 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:53.246696    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"]
Nov 03 21:14:53.253130 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:14:53.253079    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0dad2f_a03a_452b_8fba_4a20b2905abf.slice/crio-70c5d7270cb2cb9226e075ce2c8f3a34567fbd4fee1c92c93152207b942898f7 WatchSource:0}: Error finding container 70c5d7270cb2cb9226e075ce2c8f3a34567fbd4fee1c92c93152207b942898f7: Status 404 returned error can't find the container with id 70c5d7270cb2cb9226e075ce2c8f3a34567fbd4fee1c92c93152207b942898f7
Nov 03 21:14:53.333600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:53.333567    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j" event={"ID":"1b0dad2f-a03a-452b-8fba-4a20b2905abf","Type":"ContainerStarted","Data":"70c5d7270cb2cb9226e075ce2c8f3a34567fbd4fee1c92c93152207b942898f7"}
Nov 03 21:14:54.344692 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:54.344657    5196 generic.go:356] "Generic (PLEG): container finished" podID="1b0dad2f-a03a-452b-8fba-4a20b2905abf" containerID="bfe6b47675a6efdab711a10f8dd1127ab5de5fcb3b301149c1724a875be02624" exitCode=0
Nov 03 21:14:54.345179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:54.344751    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j" event={"ID":"1b0dad2f-a03a-452b-8fba-4a20b2905abf","Type":"ContainerDied","Data":"bfe6b47675a6efdab711a10f8dd1127ab5de5fcb3b301149c1724a875be02624"}
Nov 03 21:14:55.521154 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.521126    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:55.635440 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.635356    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0dad2f-a03a-452b-8fba-4a20b2905abf-kube-api-access-k7qms\" (UniqueName: \"kubernetes.io/projected/1b0dad2f-a03a-452b-8fba-4a20b2905abf-kube-api-access-k7qms\") pod \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") "
Nov 03 21:14:55.635440 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.635426    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-openshift-ca\") pod \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") "
Nov 03 21:14:55.635648 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.635460    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-xks-ca\") pod \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\" (UID: \"1b0dad2f-a03a-452b-8fba-4a20b2905abf\") "
Nov 03 21:14:55.635831 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.635804    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-openshift-ca" pod "1b0dad2f-a03a-452b-8fba-4a20b2905abf" (UID: "1b0dad2f-a03a-452b-8fba-4a20b2905abf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:55.636215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.636127    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:55.637875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.637838    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-xks-ca" pod "1b0dad2f-a03a-452b-8fba-4a20b2905abf" (UID: "1b0dad2f-a03a-452b-8fba-4a20b2905abf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:14:55.638256 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.638232    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0dad2f-a03a-452b-8fba-4a20b2905abf-kube-api-access-k7qms" pod "1b0dad2f-a03a-452b-8fba-4a20b2905abf" (UID: "1b0dad2f-a03a-452b-8fba-4a20b2905abf"). InnerVolumeSpecName "kube-api-access-k7qms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:55.737362 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.737327    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7qms\" (UniqueName: \"kubernetes.io/projected/1b0dad2f-a03a-452b-8fba-4a20b2905abf-kube-api-access-k7qms\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:55.737362 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:55.737359    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0dad2f-a03a-452b-8fba-4a20b2905abf-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.361824 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:56.361791    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"
Nov 03 21:14:56.362012 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:56.361791    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j" event={"ID":"1b0dad2f-a03a-452b-8fba-4a20b2905abf","Type":"ContainerDied","Data":"70c5d7270cb2cb9226e075ce2c8f3a34567fbd4fee1c92c93152207b942898f7"}
Nov 03 21:14:56.362012 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:56.361946    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c5d7270cb2cb9226e075ce2c8f3a34567fbd4fee1c92c93152207b942898f7"
Nov 03 21:14:57.376537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:14:57.376503    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:00.187159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.187119    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"]
Nov 03 21:15:00.187751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.187737    5196 cpu_manager.go:414] "containerMap: removing container" podUID="1b0dad2f-a03a-452b-8fba-4a20b2905abf" containerName="revoke-keys"
Nov 03 21:15:00.187796 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.187752    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0dad2f-a03a-452b-8fba-4a20b2905abf" containerName="revoke-keys"
Nov 03 21:15:00.187864 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.187856    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0dad2f-a03a-452b-8fba-4a20b2905abf" containerName="revoke-keys"
Nov 03 21:15:00.191795 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.191771    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:00.196754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.196730    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"]
Nov 03 21:15:00.200954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.200933    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:00.202298 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.202272    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"]
Nov 03 21:15:00.207268 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.207246    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:00.236220 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.236174    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"]
Nov 03 21:15:00.240605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.240576    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:00.240782 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.240691    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"]
Nov 03 21:15:00.245786 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.245759    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"]
Nov 03 21:15:00.245960 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.245894    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:00.251940 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.251905    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:00.265370 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.265040    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:15:00.298445 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.298408    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"]
Nov 03 21:15:00.302690 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.302660    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"]
Nov 03 21:15:00.308245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.307661    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49561-ad85-4e2e-b142-7fe299d71b61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq\" (UID: \"45f49561-ad85-4e2e-b142-7fe299d71b61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:00.308245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.307771    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00856d79-dc98-421c-8d49-a74b92de46e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l\" (UID: \"00856d79-dc98-421c-8d49-a74b92de46e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:00.308245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.307820    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt4cb\" (UniqueName: \"kubernetes.io/projected/01617010-30e7-4bab-8855-2a909fb4eb57-kube-api-access-tt4cb\") pod \"maas-api-key-cleanup-29895675-hvmzm\" (UID: \"01617010-30e7-4bab-8855-2a909fb4eb57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:00.308245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.307888    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5555t\" (UniqueName: \"kubernetes.io/projected/00856d79-dc98-421c-8d49-a74b92de46e8-kube-api-access-5555t\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l\" (UID: \"00856d79-dc98-421c-8d49-a74b92de46e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:00.308245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.307990    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5rt7\" (UniqueName: \"kubernetes.io/projected/45f49561-ad85-4e2e-b142-7fe299d71b61-kube-api-access-x5rt7\") pod \"maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq\" (UID: \"45f49561-ad85-4e2e-b142-7fe299d71b61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:00.308245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.308025    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01617010-30e7-4bab-8855-2a909fb4eb57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-hvmzm\" (UID: \"01617010-30e7-4bab-8855-2a909fb4eb57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:00.308726 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.308543    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"]
Nov 03 21:15:00.326236 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.326204    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"]
Nov 03 21:15:00.332202 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.332169    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"]
Nov 03 21:15:00.335944 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.335915    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"]
Nov 03 21:15:00.408903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.408839    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00856d79-dc98-421c-8d49-a74b92de46e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l\" (UID: \"00856d79-dc98-421c-8d49-a74b92de46e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:00.408903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.408909    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwpl\" (UniqueName: \"kubernetes.io/projected/43e1685f-1649-4108-b780-d02178c726e5-kube-api-access-qvwpl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n\" (UID: \"43e1685f-1649-4108-b780-d02178c726e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:00.409216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.408936    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt4cb\" (UniqueName: \"kubernetes.io/projected/01617010-30e7-4bab-8855-2a909fb4eb57-kube-api-access-tt4cb\") pod \"maas-api-key-cleanup-29895675-hvmzm\" (UID: \"01617010-30e7-4bab-8855-2a909fb4eb57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:00.409216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.408978    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1685f-1649-4108-b780-d02178c726e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n\" (UID: \"43e1685f-1649-4108-b780-d02178c726e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:00.409216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409055    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xchz8\" (UniqueName: \"kubernetes.io/projected/d78ed0bc-954a-4a1f-95f1-b3f25c404960-kube-api-access-xchz8\") pod \"maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l\" (UID: \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:00.409216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409130    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5555t\" (UniqueName: \"kubernetes.io/projected/00856d79-dc98-421c-8d49-a74b92de46e8-kube-api-access-5555t\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l\" (UID: \"00856d79-dc98-421c-8d49-a74b92de46e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:00.409216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409194    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e78536-6f08-4925-8493-e8958635e812-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb\" (UID: \"43e78536-6f08-4925-8493-e8958635e812\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:00.409507 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409292    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ed0bc-954a-4a1f-95f1-b3f25c404960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l\" (UID: \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:00.409507 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409372    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rt7\" (UniqueName: \"kubernetes.io/projected/45f49561-ad85-4e2e-b142-7fe299d71b61-kube-api-access-x5rt7\") pod \"maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq\" (UID: \"45f49561-ad85-4e2e-b142-7fe299d71b61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:00.409507 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409444    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01617010-30e7-4bab-8855-2a909fb4eb57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-hvmzm\" (UID: \"01617010-30e7-4bab-8855-2a909fb4eb57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:00.409672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409528    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49561-ad85-4e2e-b142-7fe299d71b61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq\" (UID: \"45f49561-ad85-4e2e-b142-7fe299d71b61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:00.409672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409578    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klblz\" (UniqueName: \"kubernetes.io/projected/43e78536-6f08-4925-8493-e8958635e812-kube-api-access-klblz\") pod \"maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb\" (UID: \"43e78536-6f08-4925-8493-e8958635e812\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:00.409672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.409606    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00856d79-dc98-421c-8d49-a74b92de46e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l\" (UID: \"00856d79-dc98-421c-8d49-a74b92de46e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:00.410137 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.410084    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01617010-30e7-4bab-8855-2a909fb4eb57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-hvmzm\" (UID: \"01617010-30e7-4bab-8855-2a909fb4eb57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:00.410186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.410085    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49561-ad85-4e2e-b142-7fe299d71b61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq\" (UID: \"45f49561-ad85-4e2e-b142-7fe299d71b61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:00.463708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.463632    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rt7\" (UniqueName: \"kubernetes.io/projected/45f49561-ad85-4e2e-b142-7fe299d71b61-kube-api-access-x5rt7\") pod \"maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq\" (UID: \"45f49561-ad85-4e2e-b142-7fe299d71b61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:00.463870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.463717    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt4cb\" (UniqueName: \"kubernetes.io/projected/01617010-30e7-4bab-8855-2a909fb4eb57-kube-api-access-tt4cb\") pod \"maas-api-key-cleanup-29895675-hvmzm\" (UID: \"01617010-30e7-4bab-8855-2a909fb4eb57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:00.463870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.463753    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5555t\" (UniqueName: \"kubernetes.io/projected/00856d79-dc98-421c-8d49-a74b92de46e8-kube-api-access-5555t\") pod \"maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l\" (UID: \"00856d79-dc98-421c-8d49-a74b92de46e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:00.510935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.510892    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klblz\" (UniqueName: \"kubernetes.io/projected/43e78536-6f08-4925-8493-e8958635e812-kube-api-access-klblz\") pod \"maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb\" (UID: \"43e78536-6f08-4925-8493-e8958635e812\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:00.511166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.510968    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwpl\" (UniqueName: \"kubernetes.io/projected/43e1685f-1649-4108-b780-d02178c726e5-kube-api-access-qvwpl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n\" (UID: \"43e1685f-1649-4108-b780-d02178c726e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:00.511166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.511019    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1685f-1649-4108-b780-d02178c726e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n\" (UID: \"43e1685f-1649-4108-b780-d02178c726e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:00.511166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.511046    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xchz8\" (UniqueName: \"kubernetes.io/projected/d78ed0bc-954a-4a1f-95f1-b3f25c404960-kube-api-access-xchz8\") pod \"maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l\" (UID: \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:00.511166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.511076    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e78536-6f08-4925-8493-e8958635e812-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb\" (UID: \"43e78536-6f08-4925-8493-e8958635e812\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:00.511166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.511139    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ed0bc-954a-4a1f-95f1-b3f25c404960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l\" (UID: \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:00.511515 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.511491    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:00.511791 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.511771    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1685f-1649-4108-b780-d02178c726e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n\" (UID: \"43e1685f-1649-4108-b780-d02178c726e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:00.511940 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.511918    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ed0bc-954a-4a1f-95f1-b3f25c404960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l\" (UID: \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:00.511977 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.511918    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e78536-6f08-4925-8493-e8958635e812-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb\" (UID: \"43e78536-6f08-4925-8493-e8958635e812\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:00.520382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.520355    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:00.530357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.530328    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:00.566266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.566227    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klblz\" (UniqueName: \"kubernetes.io/projected/43e78536-6f08-4925-8493-e8958635e812-kube-api-access-klblz\") pod \"maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb\" (UID: \"43e78536-6f08-4925-8493-e8958635e812\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:00.568145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.568113    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xchz8\" (UniqueName: \"kubernetes.io/projected/d78ed0bc-954a-4a1f-95f1-b3f25c404960-kube-api-access-xchz8\") pod \"maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l\" (UID: \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:00.572384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.571976    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwpl\" (UniqueName: \"kubernetes.io/projected/43e1685f-1649-4108-b780-d02178c726e5-kube-api-access-qvwpl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n\" (UID: \"43e1685f-1649-4108-b780-d02178c726e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:00.578318 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.577416    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:00.585607 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.585570    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:00.858539 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.858504    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:00.906303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.906264    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"]
Nov 03 21:15:00.911396 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:00.911300    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00856d79_dc98_421c_8d49_a74b92de46e8.slice/crio-82d8e81944e3a0a6dd88afa003e089de4a6ab0873be2fbdf69d6030cfe9ccbfa WatchSource:0}: Error finding container 82d8e81944e3a0a6dd88afa003e089de4a6ab0873be2fbdf69d6030cfe9ccbfa: Status 404 returned error can't find the container with id 82d8e81944e3a0a6dd88afa003e089de4a6ab0873be2fbdf69d6030cfe9ccbfa
Nov 03 21:15:00.958397 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.958351    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"]
Nov 03 21:15:00.964172 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:00.964104    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"]
Nov 03 21:15:00.964172 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:00.964125    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01617010_30e7_4bab_8855_2a909fb4eb57.slice/crio-5eb25a2553a60229b798312a7487ddb7890c3fea1f79da07b41996dc5a125138 WatchSource:0}: Error finding container 5eb25a2553a60229b798312a7487ddb7890c3fea1f79da07b41996dc5a125138: Status 404 returned error can't find the container with id 5eb25a2553a60229b798312a7487ddb7890c3fea1f79da07b41996dc5a125138
Nov 03 21:15:00.965240 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:00.965219    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f49561_ad85_4e2e_b142_7fe299d71b61.slice/crio-63b716aa78d8e4cf1dfa4de2097540c8bf0dba9391bf4beeeb67213f5c90be5a WatchSource:0}: Error finding container 63b716aa78d8e4cf1dfa4de2097540c8bf0dba9391bf4beeeb67213f5c90be5a: Status 404 returned error can't find the container with id 63b716aa78d8e4cf1dfa4de2097540c8bf0dba9391bf4beeeb67213f5c90be5a
Nov 03 21:15:01.015150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.015118    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"]
Nov 03 21:15:01.019319 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:01.019286    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78ed0bc_954a_4a1f_95f1_b3f25c404960.slice/crio-b91ce34c1b148f50413c00f3006da29b22c5bf93c2a4b068bff3e7aa9f6bf1a2 WatchSource:0}: Error finding container b91ce34c1b148f50413c00f3006da29b22c5bf93c2a4b068bff3e7aa9f6bf1a2: Status 404 returned error can't find the container with id b91ce34c1b148f50413c00f3006da29b22c5bf93c2a4b068bff3e7aa9f6bf1a2
Nov 03 21:15:01.031283 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.031252    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"]
Nov 03 21:15:01.037563 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:01.037533    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e78536_6f08_4925_8493_e8958635e812.slice/crio-431ebe94b3f66c796fc0f425f1b73b6b179feb5228faa53ff4b5f8acd73eba15 WatchSource:0}: Error finding container 431ebe94b3f66c796fc0f425f1b73b6b179feb5228faa53ff4b5f8acd73eba15: Status 404 returned error can't find the container with id 431ebe94b3f66c796fc0f425f1b73b6b179feb5228faa53ff4b5f8acd73eba15
Nov 03 21:15:01.244198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.244080    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"]
Nov 03 21:15:01.249189 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:01.249158    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e1685f_1649_4108_b780_d02178c726e5.slice/crio-cabe74de12a6cf91f55ac0b9ea713cb28b6e2bffa293795de6d8391a988b3fee WatchSource:0}: Error finding container cabe74de12a6cf91f55ac0b9ea713cb28b6e2bffa293795de6d8391a988b3fee: Status 404 returned error can't find the container with id cabe74de12a6cf91f55ac0b9ea713cb28b6e2bffa293795de6d8391a988b3fee
Nov 03 21:15:01.405907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.405873    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" event={"ID":"01617010-30e7-4bab-8855-2a909fb4eb57","Type":"ContainerStarted","Data":"5eb25a2553a60229b798312a7487ddb7890c3fea1f79da07b41996dc5a125138"}
Nov 03 21:15:01.407234 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.407200    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" event={"ID":"d78ed0bc-954a-4a1f-95f1-b3f25c404960","Type":"ContainerStarted","Data":"b91ce34c1b148f50413c00f3006da29b22c5bf93c2a4b068bff3e7aa9f6bf1a2"}
Nov 03 21:15:01.408473 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.408446    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" event={"ID":"43e78536-6f08-4925-8493-e8958635e812","Type":"ContainerStarted","Data":"431ebe94b3f66c796fc0f425f1b73b6b179feb5228faa53ff4b5f8acd73eba15"}
Nov 03 21:15:01.409773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.409752    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" event={"ID":"43e1685f-1649-4108-b780-d02178c726e5","Type":"ContainerStarted","Data":"cabe74de12a6cf91f55ac0b9ea713cb28b6e2bffa293795de6d8391a988b3fee"}
Nov 03 21:15:01.411061 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.411039    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" event={"ID":"00856d79-dc98-421c-8d49-a74b92de46e8","Type":"ContainerStarted","Data":"82d8e81944e3a0a6dd88afa003e089de4a6ab0873be2fbdf69d6030cfe9ccbfa"}
Nov 03 21:15:01.412715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.412695    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" event={"ID":"45f49561-ad85-4e2e-b142-7fe299d71b61","Type":"ContainerStarted","Data":"a24c48e5338827b53803f596a6af2e20df72f0e9439a56d0e607442abc33d14f"}
Nov 03 21:15:01.412815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.412729    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" event={"ID":"45f49561-ad85-4e2e-b142-7fe299d71b61","Type":"ContainerStarted","Data":"63b716aa78d8e4cf1dfa4de2097540c8bf0dba9391bf4beeeb67213f5c90be5a"}
Nov 03 21:15:01.523867 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:01.523747    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" podStartSLOduration=1.523732437 podStartE2EDuration="1.523732437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.521346775 +0000 UTC m=+2741.525305523" watchObservedRunningTime="2026-11-03 21:15:01.523732437 +0000 UTC m=+2741.527691196"
Nov 03 21:15:02.156245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.156216    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:02.424181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.424144    5196 generic.go:356] "Generic (PLEG): container finished" podID="43e1685f-1649-4108-b780-d02178c726e5" containerID="a56e1ff646f6fb5746981421667a360fdcdc7d9f6d3621d443422e42c5cd1cdd" exitCode=60
Nov 03 21:15:02.424641 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.424245    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" event={"ID":"43e1685f-1649-4108-b780-d02178c726e5","Type":"ContainerDied","Data":"a56e1ff646f6fb5746981421667a360fdcdc7d9f6d3621d443422e42c5cd1cdd"}
Nov 03 21:15:02.424641 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.424523    5196 scope.go:122] "RemoveContainer" containerID="a56e1ff646f6fb5746981421667a360fdcdc7d9f6d3621d443422e42c5cd1cdd"
Nov 03 21:15:02.426073 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.426044    5196 generic.go:356] "Generic (PLEG): container finished" podID="00856d79-dc98-421c-8d49-a74b92de46e8" containerID="efc5b91c97f7c835978ebc3c9cbd753161ab190604b13f007b74bf81b1a03741" exitCode=60
Nov 03 21:15:02.426304 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.426122    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" event={"ID":"00856d79-dc98-421c-8d49-a74b92de46e8","Type":"ContainerDied","Data":"efc5b91c97f7c835978ebc3c9cbd753161ab190604b13f007b74bf81b1a03741"}
Nov 03 21:15:02.426461 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.426445    5196 scope.go:122] "RemoveContainer" containerID="efc5b91c97f7c835978ebc3c9cbd753161ab190604b13f007b74bf81b1a03741"
Nov 03 21:15:02.428162 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.428109    5196 generic.go:356] "Generic (PLEG): container finished" podID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerID="a24c48e5338827b53803f596a6af2e20df72f0e9439a56d0e607442abc33d14f" exitCode=60
Nov 03 21:15:02.428247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.428189    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" event={"ID":"45f49561-ad85-4e2e-b142-7fe299d71b61","Type":"ContainerDied","Data":"a24c48e5338827b53803f596a6af2e20df72f0e9439a56d0e607442abc33d14f"}
Nov 03 21:15:02.428488 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.428467    5196 scope.go:122] "RemoveContainer" containerID="a24c48e5338827b53803f596a6af2e20df72f0e9439a56d0e607442abc33d14f"
Nov 03 21:15:02.430142 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.430123    5196 generic.go:356] "Generic (PLEG): container finished" podID="01617010-30e7-4bab-8855-2a909fb4eb57" containerID="e46764398252999f83b6055208faaa347aa766a177965cf8106fdf4d9b99f14d" exitCode=60
Nov 03 21:15:02.430240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.430156    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" event={"ID":"01617010-30e7-4bab-8855-2a909fb4eb57","Type":"ContainerDied","Data":"e46764398252999f83b6055208faaa347aa766a177965cf8106fdf4d9b99f14d"}
Nov 03 21:15:02.430429 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.430412    5196 scope.go:122] "RemoveContainer" containerID="e46764398252999f83b6055208faaa347aa766a177965cf8106fdf4d9b99f14d"
Nov 03 21:15:02.431860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.431838    5196 generic.go:356] "Generic (PLEG): container finished" podID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerID="8d143094fffbbcae85bdfa564fa613aa611613408a7b70f7d6b93445376cc5e0" exitCode=60
Nov 03 21:15:02.431959 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.431922    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" event={"ID":"d78ed0bc-954a-4a1f-95f1-b3f25c404960","Type":"ContainerDied","Data":"8d143094fffbbcae85bdfa564fa613aa611613408a7b70f7d6b93445376cc5e0"}
Nov 03 21:15:02.432153 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.432138    5196 scope.go:122] "RemoveContainer" containerID="8d143094fffbbcae85bdfa564fa613aa611613408a7b70f7d6b93445376cc5e0"
Nov 03 21:15:02.433735 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.433716    5196 generic.go:356] "Generic (PLEG): container finished" podID="43e78536-6f08-4925-8493-e8958635e812" containerID="82d0a0d04c28a08c323aca6c23b2d9a3598ebd22978ea55c4941176f03e4fe4a" exitCode=60
Nov 03 21:15:02.433819 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.433784    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" event={"ID":"43e78536-6f08-4925-8493-e8958635e812","Type":"ContainerDied","Data":"82d0a0d04c28a08c323aca6c23b2d9a3598ebd22978ea55c4941176f03e4fe4a"}
Nov 03 21:15:02.433954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:02.433942    5196 scope.go:122] "RemoveContainer" containerID="82d0a0d04c28a08c323aca6c23b2d9a3598ebd22978ea55c4941176f03e4fe4a"
Nov 03 21:15:03.447412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.447374    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" event={"ID":"43e1685f-1649-4108-b780-d02178c726e5","Type":"ContainerStarted","Data":"d0b7a00de627f2f93d06d508e11867de2178423bb5356f7527b57412cb33e2f2"}
Nov 03 21:15:03.449881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.449850    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" event={"ID":"00856d79-dc98-421c-8d49-a74b92de46e8","Type":"ContainerStarted","Data":"dcc40793f40407cfc36b6103ba7157f0f1719c83cb5db8c4b4c80b0865f86555"}
Nov 03 21:15:03.451986 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.451960    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" event={"ID":"45f49561-ad85-4e2e-b142-7fe299d71b61","Type":"ContainerStarted","Data":"0f4fd92818fffb986126ef5245f4038f34008a3a602fc1469e53441a9474693c"}
Nov 03 21:15:03.454125 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.454068    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" event={"ID":"01617010-30e7-4bab-8855-2a909fb4eb57","Type":"ContainerStarted","Data":"5358336cba5bd443e1d196dc0be568ebc3d2c522c1595c47b663722cd0189720"}
Nov 03 21:15:03.457222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.457197    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" event={"ID":"d78ed0bc-954a-4a1f-95f1-b3f25c404960","Type":"ContainerStarted","Data":"4d8f7c9217bd67e52d31a58f564d1c63c7bc0af6aa9d25284d914dd04d25ea74"}
Nov 03 21:15:03.459192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.459071    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" event={"ID":"43e78536-6f08-4925-8493-e8958635e812","Type":"ContainerStarted","Data":"02fb9d5f90c45d48eb98183682b012f2fadf2025e56e17781485a41a3efc2b7d"}
Nov 03 21:15:03.459414 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.459397    5196 scope.go:122] "RemoveContainer" containerID="02fb9d5f90c45d48eb98183682b012f2fadf2025e56e17781485a41a3efc2b7d"
Nov 03 21:15:03.459649 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:03.459628    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb_odh-ai-gateway-infra(43e78536-6f08-4925-8493-e8958635e812)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" podUID="43e78536-6f08-4925-8493-e8958635e812"
Nov 03 21:15:03.555424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.555369    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" podStartSLOduration=3.555354394 podStartE2EDuration="3.555354394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.55417709 +0000 UTC m=+2743.558135833" watchObservedRunningTime="2026-11-03 21:15:03.555354394 +0000 UTC m=+2743.559313118"
Nov 03 21:15:03.663048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.662989    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" podStartSLOduration=3.6629722989999998 podStartE2EDuration="3.662972299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.660652486 +0000 UTC m=+2743.664611238" watchObservedRunningTime="2026-11-03 21:15:03.662972299 +0000 UTC m=+2743.666931023"
Nov 03 21:15:03.995292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:03.995180    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" podStartSLOduration=3.995163924 podStartE2EDuration="3.995163924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.993675879 +0000 UTC m=+2743.997634631" watchObservedRunningTime="2026-11-03 21:15:03.995163924 +0000 UTC m=+2743.999122648"
Nov 03 21:15:04.101215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.101149    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" podStartSLOduration=4.10113234 podStartE2EDuration="4.10113234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:04.09943026 +0000 UTC m=+2744.103388981" watchObservedRunningTime="2026-11-03 21:15:04.10113234 +0000 UTC m=+2744.105091065"
Nov 03 21:15:04.470490 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.470454    5196 generic.go:356] "Generic (PLEG): container finished" podID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerID="0f4fd92818fffb986126ef5245f4038f34008a3a602fc1469e53441a9474693c" exitCode=60
Nov 03 21:15:04.470939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.470545    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" event={"ID":"45f49561-ad85-4e2e-b142-7fe299d71b61","Type":"ContainerDied","Data":"0f4fd92818fffb986126ef5245f4038f34008a3a602fc1469e53441a9474693c"}
Nov 03 21:15:04.470939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.470623    5196 scope.go:122] "RemoveContainer" containerID="a24c48e5338827b53803f596a6af2e20df72f0e9439a56d0e607442abc33d14f"
Nov 03 21:15:04.470939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.470832    5196 scope.go:122] "RemoveContainer" containerID="0f4fd92818fffb986126ef5245f4038f34008a3a602fc1469e53441a9474693c"
Nov 03 21:15:04.471197 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:04.471169    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq_odh-ai-gateway-infra(45f49561-ad85-4e2e-b142-7fe299d71b61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61"
Nov 03 21:15:04.472841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.472818    5196 generic.go:356] "Generic (PLEG): container finished" podID="01617010-30e7-4bab-8855-2a909fb4eb57" containerID="5358336cba5bd443e1d196dc0be568ebc3d2c522c1595c47b663722cd0189720" exitCode=60
Nov 03 21:15:04.472952 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.472902    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" event={"ID":"01617010-30e7-4bab-8855-2a909fb4eb57","Type":"ContainerDied","Data":"5358336cba5bd443e1d196dc0be568ebc3d2c522c1595c47b663722cd0189720"}
Nov 03 21:15:04.473160 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.473144    5196 scope.go:122] "RemoveContainer" containerID="5358336cba5bd443e1d196dc0be568ebc3d2c522c1595c47b663722cd0189720"
Nov 03 21:15:04.473432 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:04.473410    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-hvmzm_odh-ai-gateway-infra(01617010-30e7-4bab-8855-2a909fb4eb57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" podUID="01617010-30e7-4bab-8855-2a909fb4eb57"
Nov 03 21:15:04.474935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.474912    5196 generic.go:356] "Generic (PLEG): container finished" podID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerID="4d8f7c9217bd67e52d31a58f564d1c63c7bc0af6aa9d25284d914dd04d25ea74" exitCode=60
Nov 03 21:15:04.475036 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.474989    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" event={"ID":"d78ed0bc-954a-4a1f-95f1-b3f25c404960","Type":"ContainerDied","Data":"4d8f7c9217bd67e52d31a58f564d1c63c7bc0af6aa9d25284d914dd04d25ea74"}
Nov 03 21:15:04.475230 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.475215    5196 scope.go:122] "RemoveContainer" containerID="4d8f7c9217bd67e52d31a58f564d1c63c7bc0af6aa9d25284d914dd04d25ea74"
Nov 03 21:15:04.475447 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:04.475425    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l_odh-ai-gateway-infra(d78ed0bc-954a-4a1f-95f1-b3f25c404960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960"
Nov 03 21:15:04.477081 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.477060    5196 generic.go:356] "Generic (PLEG): container finished" podID="43e78536-6f08-4925-8493-e8958635e812" containerID="02fb9d5f90c45d48eb98183682b012f2fadf2025e56e17781485a41a3efc2b7d" exitCode=60
Nov 03 21:15:04.477271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.477141    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" event={"ID":"43e78536-6f08-4925-8493-e8958635e812","Type":"ContainerDied","Data":"02fb9d5f90c45d48eb98183682b012f2fadf2025e56e17781485a41a3efc2b7d"}
Nov 03 21:15:04.477404 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.477390    5196 scope.go:122] "RemoveContainer" containerID="02fb9d5f90c45d48eb98183682b012f2fadf2025e56e17781485a41a3efc2b7d"
Nov 03 21:15:04.477608 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:04.477587    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb_odh-ai-gateway-infra(43e78536-6f08-4925-8493-e8958635e812)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" podUID="43e78536-6f08-4925-8493-e8958635e812"
Nov 03 21:15:04.479327 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.479308    5196 generic.go:356] "Generic (PLEG): container finished" podID="43e1685f-1649-4108-b780-d02178c726e5" containerID="d0b7a00de627f2f93d06d508e11867de2178423bb5356f7527b57412cb33e2f2" exitCode=60
Nov 03 21:15:04.479444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.479367    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" event={"ID":"43e1685f-1649-4108-b780-d02178c726e5","Type":"ContainerDied","Data":"d0b7a00de627f2f93d06d508e11867de2178423bb5356f7527b57412cb33e2f2"}
Nov 03 21:15:04.479611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.479594    5196 scope.go:122] "RemoveContainer" containerID="d0b7a00de627f2f93d06d508e11867de2178423bb5356f7527b57412cb33e2f2"
Nov 03 21:15:04.479817 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:04.479798    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n_odh-ai-gateway-infra(43e1685f-1649-4108-b780-d02178c726e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" podUID="43e1685f-1649-4108-b780-d02178c726e5"
Nov 03 21:15:04.481833 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.481812    5196 generic.go:356] "Generic (PLEG): container finished" podID="00856d79-dc98-421c-8d49-a74b92de46e8" containerID="dcc40793f40407cfc36b6103ba7157f0f1719c83cb5db8c4b4c80b0865f86555" exitCode=60
Nov 03 21:15:04.481930 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.481897    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" event={"ID":"00856d79-dc98-421c-8d49-a74b92de46e8","Type":"ContainerDied","Data":"dcc40793f40407cfc36b6103ba7157f0f1719c83cb5db8c4b4c80b0865f86555"}
Nov 03 21:15:04.482061 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.482048    5196 scope.go:122] "RemoveContainer" containerID="dcc40793f40407cfc36b6103ba7157f0f1719c83cb5db8c4b4c80b0865f86555"
Nov 03 21:15:04.482271 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:04.482255    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l_odh-ai-gateway-infra(00856d79-dc98-421c-8d49-a74b92de46e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" podUID="00856d79-dc98-421c-8d49-a74b92de46e8"
Nov 03 21:15:04.485757 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.485735    5196 scope.go:122] "RemoveContainer" containerID="e46764398252999f83b6055208faaa347aa766a177965cf8106fdf4d9b99f14d"
Nov 03 21:15:04.500361 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.500339    5196 scope.go:122] "RemoveContainer" containerID="8d143094fffbbcae85bdfa564fa613aa611613408a7b70f7d6b93445376cc5e0"
Nov 03 21:15:04.514487 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.514462    5196 scope.go:122] "RemoveContainer" containerID="82d0a0d04c28a08c323aca6c23b2d9a3598ebd22978ea55c4941176f03e4fe4a"
Nov 03 21:15:04.529573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.529551    5196 scope.go:122] "RemoveContainer" containerID="a56e1ff646f6fb5746981421667a360fdcdc7d9f6d3621d443422e42c5cd1cdd"
Nov 03 21:15:04.544346 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:04.544321    5196 scope.go:122] "RemoveContainer" containerID="efc5b91c97f7c835978ebc3c9cbd753161ab190604b13f007b74bf81b1a03741"
Nov 03 21:15:05.492222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:05.492189    5196 scope.go:122] "RemoveContainer" containerID="0f4fd92818fffb986126ef5245f4038f34008a3a602fc1469e53441a9474693c"
Nov 03 21:15:05.492800 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:05.492570    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq_odh-ai-gateway-infra(45f49561-ad85-4e2e-b142-7fe299d71b61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61"
Nov 03 21:15:05.493950 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:05.493928    5196 scope.go:122] "RemoveContainer" containerID="5358336cba5bd443e1d196dc0be568ebc3d2c522c1595c47b663722cd0189720"
Nov 03 21:15:05.494278 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:05.494256    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-hvmzm_odh-ai-gateway-infra(01617010-30e7-4bab-8855-2a909fb4eb57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" podUID="01617010-30e7-4bab-8855-2a909fb4eb57"
Nov 03 21:15:05.495691 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:05.495671    5196 scope.go:122] "RemoveContainer" containerID="4d8f7c9217bd67e52d31a58f564d1c63c7bc0af6aa9d25284d914dd04d25ea74"
Nov 03 21:15:05.495950 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:05.495929    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l_odh-ai-gateway-infra(d78ed0bc-954a-4a1f-95f1-b3f25c404960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960"
Nov 03 21:15:05.497396 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:05.497374    5196 scope.go:122] "RemoveContainer" containerID="02fb9d5f90c45d48eb98183682b012f2fadf2025e56e17781485a41a3efc2b7d"
Nov 03 21:15:05.497683 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:05.497651    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb_odh-ai-gateway-infra(43e78536-6f08-4925-8493-e8958635e812)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" podUID="43e78536-6f08-4925-8493-e8958635e812"
Nov 03 21:15:05.499236 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:05.499218    5196 scope.go:122] "RemoveContainer" containerID="d0b7a00de627f2f93d06d508e11867de2178423bb5356f7527b57412cb33e2f2"
Nov 03 21:15:05.499490 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:05.499471    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n_odh-ai-gateway-infra(43e1685f-1649-4108-b780-d02178c726e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" podUID="43e1685f-1649-4108-b780-d02178c726e5"
Nov 03 21:15:05.500989 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:05.500972    5196 scope.go:122] "RemoveContainer" containerID="dcc40793f40407cfc36b6103ba7157f0f1719c83cb5db8c4b4c80b0865f86555"
Nov 03 21:15:05.501210 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:05.501190    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l_odh-ai-gateway-infra(00856d79-dc98-421c-8d49-a74b92de46e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" podUID="00856d79-dc98-421c-8d49-a74b92de46e8"
Nov 03 21:15:06.556786 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.556747    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx" podUID="ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" containerName="main" containerID="cri-o://515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55" gracePeriod=45
Nov 03 21:15:06.642923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.642880    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx" podUID="ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.240:8000/ready\": dial tcp 10.132.0.240:8000: connect: connection refused"
Nov 03 21:15:06.850920 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.850893    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:15:06.900679 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.900647    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-home\") pod \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") "
Nov 03 21:15:06.900896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.900757    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-dshm\") pod \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") "
Nov 03 21:15:06.900896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.900788    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-model-cache\") pod \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") "
Nov 03 21:15:06.900896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.900889    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tmp-dir\") pod \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") "
Nov 03 21:15:06.901086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.900936    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tls-certs\") pod \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") "
Nov 03 21:15:06.901086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.900947    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-home" pod "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" (UID: "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:06.901086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.901078    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-kube-api-access-x2jdx\" (UniqueName: \"kubernetes.io/projected/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-kube-api-access-x2jdx\") pod \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\" (UID: \"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7\") "
Nov 03 21:15:06.901267 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.901158    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-model-cache" pod "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" (UID: "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:06.901267 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.901171    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tmp-dir" pod "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" (UID: "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:06.901562 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.901536    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:06.901562 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.901566    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:06.901740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.901579    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:06.903323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.903304    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-dshm" pod "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" (UID: "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:06.903436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.903329    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tls-certs" pod "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" (UID: "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:06.903436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.903355    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-kube-api-access-x2jdx" pod "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" (UID: "ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7"). InnerVolumeSpecName "kube-api-access-x2jdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:06.964086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:06.964033    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:07.003021 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.002979    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:07.003021 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.003023    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:07.003021 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.003040    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jdx\" (UniqueName: \"kubernetes.io/projected/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7-kube-api-access-x2jdx\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:07.517486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.517444    5196 generic.go:356] "Generic (PLEG): container finished" podID="ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" containerID="515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55" exitCode=0
Nov 03 21:15:07.517695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.517498    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"
Nov 03 21:15:07.517695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.517529    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx" event={"ID":"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7","Type":"ContainerDied","Data":"515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55"}
Nov 03 21:15:07.517695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.517576    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx" event={"ID":"ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7","Type":"ContainerDied","Data":"c3ab961269c63a07f50e0d34270870777a83c013d34b5e3df3de3ec9183d4ada"}
Nov 03 21:15:07.517695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.517599    5196 scope.go:122] "RemoveContainer" containerID="515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55"
Nov 03 21:15:07.529597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.529578    5196 scope.go:122] "RemoveContainer" containerID="515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55"
Nov 03 21:15:07.529927 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:07.529910    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55\": container with ID starting with 515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55 not found: ID does not exist" containerID="515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55"
Nov 03 21:15:07.530012 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.529938    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55"} err="failed to get container status \"515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55\": rpc error: code = NotFound desc = could not find container \"515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55\": container with ID starting with 515aeedd957fa5e3a93aedf066b1d06c38c4fec072fc2c981267e5b9dc7aaf55 not found: ID does not exist"
Nov 03 21:15:07.673337 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.673300    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"]
Nov 03 21:15:07.720052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:07.720016    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7c34ff/test-model-7c34ff-kserve-c4d79d5f7-k8qsx"]
Nov 03 21:15:08.110244 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:08.110213    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" path="/var/lib/kubelet/pods/ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7/volumes"
Nov 03 21:15:10.046854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:10.046817    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"]
Nov 03 21:15:10.094284 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:10.094242    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29c463e0-jcwt2"]
Nov 03 21:15:10.098492 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:10.098444    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"]
Nov 03 21:15:10.109544 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:10.109515    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aee95c0-42b1-4005-acec-4ef171f602cc" path="/var/lib/kubelet/pods/2aee95c0-42b1-4005-acec-4ef171f602cc/volumes"
Nov 03 21:15:10.109854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:10.109841    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"]
Nov 03 21:15:10.153604 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:10.153554    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-e0c1753c-6dgcf\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="33c2834e-5f21-4460-a456-856a6df586b3" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c1753c-6dgcf"
Nov 03 21:15:10.649977 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:10.649936    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:11.225361 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.225321    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"]
Nov 03 21:15:11.226056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.226039    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" containerName="main"
Nov 03 21:15:11.226149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.226059    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" containerName="main"
Nov 03 21:15:11.226255 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.226242    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb9bd17-20c5-428f-bb2f-e222f9c5e1e7" containerName="main"
Nov 03 21:15:11.229809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.229790    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.252546 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.252512    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.252687 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.252636    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6fh\" (UniqueName: \"kubernetes.io/projected/7789434b-0951-4f26-b333-ad9fded40ecc-kube-api-access-ms6fh\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.252743 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.252700    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.278997 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.278961    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:15:11.329987 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.329954    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"]
Nov 03 21:15:11.354240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.354206    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6fh\" (UniqueName: \"kubernetes.io/projected/7789434b-0951-4f26-b333-ad9fded40ecc-kube-api-access-ms6fh\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.354434 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.354275    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.354434 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.354404    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.354989 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.354959    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.357159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.357135    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.408062 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.408021    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6fh\" (UniqueName: \"kubernetes.io/projected/7789434b-0951-4f26-b333-ad9fded40ecc-kube-api-access-ms6fh\") pod \"maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.544624 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.544530    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:11.849225 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.849173    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:11.921600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:11.921564    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"]
Nov 03 21:15:11.927885 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:11.927856    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7789434b_0951_4f26_b333_ad9fded40ecc.slice/crio-6748f918a085cdb2650d4de43c549e6ef3ea75e90579f33a653516db4cddb0a6 WatchSource:0}: Error finding container 6748f918a085cdb2650d4de43c549e6ef3ea75e90579f33a653516db4cddb0a6: Status 404 returned error can't find the container with id 6748f918a085cdb2650d4de43c549e6ef3ea75e90579f33a653516db4cddb0a6
Nov 03 21:15:12.110867 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:12.110789    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c2834e-5f21-4460-a456-856a6df586b3" path="/var/lib/kubelet/pods/33c2834e-5f21-4460-a456-856a6df586b3/volumes"
Nov 03 21:15:12.561145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:12.561043    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8" event={"ID":"7789434b-0951-4f26-b333-ad9fded40ecc","Type":"ContainerStarted","Data":"2cd1dcaf42ce8b3bbe5f8c2080b48cb0676404977170ce15156b30aa632a6e1f"}
Nov 03 21:15:12.561145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:12.561108    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8" event={"ID":"7789434b-0951-4f26-b333-ad9fded40ecc","Type":"ContainerStarted","Data":"6748f918a085cdb2650d4de43c549e6ef3ea75e90579f33a653516db4cddb0a6"}
Nov 03 21:15:12.672752 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:12.672681    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8" podStartSLOduration=1.672667669 podStartE2EDuration="1.672667669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:12.668891183 +0000 UTC m=+2752.672849907" watchObservedRunningTime="2026-11-03 21:15:12.672667669 +0000 UTC m=+2752.676626392"
Nov 03 21:15:13.570342 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:13.570303    5196 generic.go:356] "Generic (PLEG): container finished" podID="7789434b-0951-4f26-b333-ad9fded40ecc" containerID="2cd1dcaf42ce8b3bbe5f8c2080b48cb0676404977170ce15156b30aa632a6e1f" exitCode=0
Nov 03 21:15:13.570800 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:13.570395    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8" event={"ID":"7789434b-0951-4f26-b333-ad9fded40ecc","Type":"ContainerDied","Data":"2cd1dcaf42ce8b3bbe5f8c2080b48cb0676404977170ce15156b30aa632a6e1f"}
Nov 03 21:15:14.102068 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.102035    5196 scope.go:122] "RemoveContainer" containerID="d0b7a00de627f2f93d06d508e11867de2178423bb5356f7527b57412cb33e2f2"
Nov 03 21:15:14.580908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.580870    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" event={"ID":"43e1685f-1649-4108-b780-d02178c726e5","Type":"ContainerStarted","Data":"c693d87204bb7b00fe5c4f9d6810d6564544014d38c36e573d112c4403c56262"}
Nov 03 21:15:14.744671 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.744642    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:14.794506 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.794464    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-xks-ca\") pod \"7789434b-0951-4f26-b333-ad9fded40ecc\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") "
Nov 03 21:15:14.794720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.794571    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-openshift-ca\") pod \"7789434b-0951-4f26-b333-ad9fded40ecc\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") "
Nov 03 21:15:14.794720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.794642    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7789434b-0951-4f26-b333-ad9fded40ecc-kube-api-access-ms6fh\" (UniqueName: \"kubernetes.io/projected/7789434b-0951-4f26-b333-ad9fded40ecc-kube-api-access-ms6fh\") pod \"7789434b-0951-4f26-b333-ad9fded40ecc\" (UID: \"7789434b-0951-4f26-b333-ad9fded40ecc\") "
Nov 03 21:15:14.794955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.794924    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-openshift-ca" pod "7789434b-0951-4f26-b333-ad9fded40ecc" (UID: "7789434b-0951-4f26-b333-ad9fded40ecc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:14.797035 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.797007    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-xks-ca" pod "7789434b-0951-4f26-b333-ad9fded40ecc" (UID: "7789434b-0951-4f26-b333-ad9fded40ecc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:14.797192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.797054    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7789434b-0951-4f26-b333-ad9fded40ecc-kube-api-access-ms6fh" pod "7789434b-0951-4f26-b333-ad9fded40ecc" (UID: "7789434b-0951-4f26-b333-ad9fded40ecc"). InnerVolumeSpecName "kube-api-access-ms6fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:14.895675 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.895644    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:14.895675 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.895672    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms6fh\" (UniqueName: \"kubernetes.io/projected/7789434b-0951-4f26-b333-ad9fded40ecc-kube-api-access-ms6fh\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:14.895675 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:14.895682    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7789434b-0951-4f26-b333-ad9fded40ecc-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:15.101411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.101374    5196 scope.go:122] "RemoveContainer" containerID="02fb9d5f90c45d48eb98183682b012f2fadf2025e56e17781485a41a3efc2b7d"
Nov 03 21:15:15.101634 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.101515    5196 scope.go:122] "RemoveContainer" containerID="5358336cba5bd443e1d196dc0be568ebc3d2c522c1595c47b663722cd0189720"
Nov 03 21:15:15.591048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.591005    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" event={"ID":"01617010-30e7-4bab-8855-2a909fb4eb57","Type":"ContainerStarted","Data":"60479dbc5054264bfa552fd7ccc76e8f9f535fc7509a4dd8e73952d1211c1855"}
Nov 03 21:15:15.593166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.593134    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" event={"ID":"43e78536-6f08-4925-8493-e8958635e812","Type":"ContainerStarted","Data":"721edef18953934eb100248fb2d77f7256bc18dc6d71dc67c3c6c2214b6f7cdc"}
Nov 03 21:15:15.594633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.594606    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8" event={"ID":"7789434b-0951-4f26-b333-ad9fded40ecc","Type":"ContainerDied","Data":"6748f918a085cdb2650d4de43c549e6ef3ea75e90579f33a653516db4cddb0a6"}
Nov 03 21:15:15.594633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.594624    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"
Nov 03 21:15:15.594800 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.594656    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6748f918a085cdb2650d4de43c549e6ef3ea75e90579f33a653516db4cddb0a6"
Nov 03 21:15:15.596713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.596688    5196 generic.go:356] "Generic (PLEG): container finished" podID="43e1685f-1649-4108-b780-d02178c726e5" containerID="c693d87204bb7b00fe5c4f9d6810d6564544014d38c36e573d112c4403c56262" exitCode=60
Nov 03 21:15:15.596846 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.596749    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" event={"ID":"43e1685f-1649-4108-b780-d02178c726e5","Type":"ContainerDied","Data":"c693d87204bb7b00fe5c4f9d6810d6564544014d38c36e573d112c4403c56262"}
Nov 03 21:15:15.596846 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.596826    5196 scope.go:122] "RemoveContainer" containerID="d0b7a00de627f2f93d06d508e11867de2178423bb5356f7527b57412cb33e2f2"
Nov 03 21:15:15.597221 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.597202    5196 scope.go:122] "RemoveContainer" containerID="c693d87204bb7b00fe5c4f9d6810d6564544014d38c36e573d112c4403c56262"
Nov 03 21:15:15.597527 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:15.597507    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n_odh-ai-gateway-infra(43e1685f-1649-4108-b780-d02178c726e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" podUID="43e1685f-1649-4108-b780-d02178c726e5"
Nov 03 21:15:15.698059 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.697955    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"]
Nov 03 21:15:15.917430 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:15.917369    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" podStartSLOduration=15.917353407 podStartE2EDuration="15.917353407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:15.914914707 +0000 UTC m=+2755.918873442" watchObservedRunningTime="2026-11-03 21:15:15.917353407 +0000 UTC m=+2755.921312145"
Nov 03 21:15:16.071697 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.071662    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:15:16.101973 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.101938    5196 scope.go:122] "RemoveContainer" containerID="dcc40793f40407cfc36b6103ba7157f0f1719c83cb5db8c4b4c80b0865f86555"
Nov 03 21:15:16.102190 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.102069    5196 scope.go:122] "RemoveContainer" containerID="4d8f7c9217bd67e52d31a58f564d1c63c7bc0af6aa9d25284d914dd04d25ea74"
Nov 03 21:15:16.620958 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.620928    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" event={"ID":"00856d79-dc98-421c-8d49-a74b92de46e8","Type":"ContainerStarted","Data":"b77066ad8353d746f07adbde06e8e7c56a93fa23510e361d231478ee5a1e069f"}
Nov 03 21:15:16.623312 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.623286    5196 generic.go:356] "Generic (PLEG): container finished" podID="01617010-30e7-4bab-8855-2a909fb4eb57" containerID="60479dbc5054264bfa552fd7ccc76e8f9f535fc7509a4dd8e73952d1211c1855" exitCode=60
Nov 03 21:15:16.623414 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.623349    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" event={"ID":"01617010-30e7-4bab-8855-2a909fb4eb57","Type":"ContainerDied","Data":"60479dbc5054264bfa552fd7ccc76e8f9f535fc7509a4dd8e73952d1211c1855"}
Nov 03 21:15:16.623414 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.623404    5196 scope.go:122] "RemoveContainer" containerID="5358336cba5bd443e1d196dc0be568ebc3d2c522c1595c47b663722cd0189720"
Nov 03 21:15:16.623775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.623753    5196 scope.go:122] "RemoveContainer" containerID="60479dbc5054264bfa552fd7ccc76e8f9f535fc7509a4dd8e73952d1211c1855"
Nov 03 21:15:16.624172 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:16.624151    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-hvmzm_odh-ai-gateway-infra(01617010-30e7-4bab-8855-2a909fb4eb57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" podUID="01617010-30e7-4bab-8855-2a909fb4eb57"
Nov 03 21:15:16.626951 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.626915    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" event={"ID":"d78ed0bc-954a-4a1f-95f1-b3f25c404960","Type":"ContainerStarted","Data":"0b08f4d68624e5e2bfd7b15b2809bea92e2d4b2d8b120fdd24023a7a0e1d2892"}
Nov 03 21:15:16.629843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.629813    5196 generic.go:356] "Generic (PLEG): container finished" podID="43e78536-6f08-4925-8493-e8958635e812" containerID="721edef18953934eb100248fb2d77f7256bc18dc6d71dc67c3c6c2214b6f7cdc" exitCode=60
Nov 03 21:15:16.629966 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.629872    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" event={"ID":"43e78536-6f08-4925-8493-e8958635e812","Type":"ContainerDied","Data":"721edef18953934eb100248fb2d77f7256bc18dc6d71dc67c3c6c2214b6f7cdc"}
Nov 03 21:15:16.630271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.630254    5196 scope.go:122] "RemoveContainer" containerID="721edef18953934eb100248fb2d77f7256bc18dc6d71dc67c3c6c2214b6f7cdc"
Nov 03 21:15:16.630584 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:16.630558    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb_odh-ai-gateway-infra(43e78536-6f08-4925-8493-e8958635e812)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" podUID="43e78536-6f08-4925-8493-e8958635e812"
Nov 03 21:15:16.690837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.690811    5196 scope.go:122] "RemoveContainer" containerID="02fb9d5f90c45d48eb98183682b012f2fadf2025e56e17781485a41a3efc2b7d"
Nov 03 21:15:16.711060 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.711027    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"]
Nov 03 21:15:16.813913 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.813888    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:16.931352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.931264    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e1685f-1649-4108-b780-d02178c726e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1685f-1649-4108-b780-d02178c726e5-maas-api-service-ca\") pod \"43e1685f-1649-4108-b780-d02178c726e5\" (UID: \"43e1685f-1649-4108-b780-d02178c726e5\") "
Nov 03 21:15:16.931524 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.931357    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e1685f-1649-4108-b780-d02178c726e5-kube-api-access-qvwpl\" (UniqueName: \"kubernetes.io/projected/43e1685f-1649-4108-b780-d02178c726e5-kube-api-access-qvwpl\") pod \"43e1685f-1649-4108-b780-d02178c726e5\" (UID: \"43e1685f-1649-4108-b780-d02178c726e5\") "
Nov 03 21:15:16.931656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.931634    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e1685f-1649-4108-b780-d02178c726e5-maas-api-service-ca" pod "43e1685f-1649-4108-b780-d02178c726e5" (UID: "43e1685f-1649-4108-b780-d02178c726e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:16.931816 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.931800    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1685f-1649-4108-b780-d02178c726e5-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:16.933909 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.933880    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e1685f-1649-4108-b780-d02178c726e5-kube-api-access-qvwpl" pod "43e1685f-1649-4108-b780-d02178c726e5" (UID: "43e1685f-1649-4108-b780-d02178c726e5"). InnerVolumeSpecName "kube-api-access-qvwpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:16.938007 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:16.937982    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"]
Nov 03 21:15:17.032837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.032800    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvwpl\" (UniqueName: \"kubernetes.io/projected/43e1685f-1649-4108-b780-d02178c726e5-kube-api-access-qvwpl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.094335 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.094301    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:17.102053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.102020    5196 scope.go:122] "RemoveContainer" containerID="0f4fd92818fffb986126ef5245f4038f34008a3a602fc1469e53441a9474693c"
Nov 03 21:15:17.641123 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.641068    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n" event={"ID":"43e1685f-1649-4108-b780-d02178c726e5","Type":"ContainerDied","Data":"cabe74de12a6cf91f55ac0b9ea713cb28b6e2bffa293795de6d8391a988b3fee"}
Nov 03 21:15:17.641617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.641146    5196 scope.go:122] "RemoveContainer" containerID="c693d87204bb7b00fe5c4f9d6810d6564544014d38c36e573d112c4403c56262"
Nov 03 21:15:17.641617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.641075    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"
Nov 03 21:15:17.643713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.643682    5196 generic.go:356] "Generic (PLEG): container finished" podID="00856d79-dc98-421c-8d49-a74b92de46e8" containerID="b77066ad8353d746f07adbde06e8e7c56a93fa23510e361d231478ee5a1e069f" exitCode=60
Nov 03 21:15:17.643868 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.643753    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" event={"ID":"00856d79-dc98-421c-8d49-a74b92de46e8","Type":"ContainerDied","Data":"b77066ad8353d746f07adbde06e8e7c56a93fa23510e361d231478ee5a1e069f"}
Nov 03 21:15:17.644223 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.644201    5196 scope.go:122] "RemoveContainer" containerID="b77066ad8353d746f07adbde06e8e7c56a93fa23510e361d231478ee5a1e069f"
Nov 03 21:15:17.644582 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:17.644562    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l_odh-ai-gateway-infra(00856d79-dc98-421c-8d49-a74b92de46e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" podUID="00856d79-dc98-421c-8d49-a74b92de46e8"
Nov 03 21:15:17.646570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.646546    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" event={"ID":"45f49561-ad85-4e2e-b142-7fe299d71b61","Type":"ContainerStarted","Data":"633ef35fa4371b1691b50fe9af7b7bffc56e997241bf12335e4d7d57382955ce"}
Nov 03 21:15:17.651852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.651829    5196 generic.go:356] "Generic (PLEG): container finished" podID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerID="0b08f4d68624e5e2bfd7b15b2809bea92e2d4b2d8b120fdd24023a7a0e1d2892" exitCode=60
Nov 03 21:15:17.651963 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.651923    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" event={"ID":"d78ed0bc-954a-4a1f-95f1-b3f25c404960","Type":"ContainerDied","Data":"0b08f4d68624e5e2bfd7b15b2809bea92e2d4b2d8b120fdd24023a7a0e1d2892"}
Nov 03 21:15:17.652268 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.652252    5196 scope.go:122] "RemoveContainer" containerID="0b08f4d68624e5e2bfd7b15b2809bea92e2d4b2d8b120fdd24023a7a0e1d2892"
Nov 03 21:15:17.652560 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:17.652541    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l_odh-ai-gateway-infra(d78ed0bc-954a-4a1f-95f1-b3f25c404960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960"
Nov 03 21:15:17.654824 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.654774    5196 scope.go:122] "RemoveContainer" containerID="dcc40793f40407cfc36b6103ba7157f0f1719c83cb5db8c4b4c80b0865f86555"
Nov 03 21:15:17.670632 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.670589    5196 scope.go:122] "RemoveContainer" containerID="4d8f7c9217bd67e52d31a58f564d1c63c7bc0af6aa9d25284d914dd04d25ea74"
Nov 03 21:15:17.800321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.800241    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"]
Nov 03 21:15:17.806573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.806531    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1e862b-29895675-8vc6n"]
Nov 03 21:15:17.836910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.836880    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:17.855365 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.855336    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:17.865182 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.865150    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"]
Nov 03 21:15:17.942840 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.942806    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e78536-6f08-4925-8493-e8958635e812-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e78536-6f08-4925-8493-e8958635e812-maas-api-service-ca\") pod \"43e78536-6f08-4925-8493-e8958635e812\" (UID: \"43e78536-6f08-4925-8493-e8958635e812\") "
Nov 03 21:15:17.943031 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.942848    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01617010-30e7-4bab-8855-2a909fb4eb57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01617010-30e7-4bab-8855-2a909fb4eb57-maas-api-service-ca\") pod \"01617010-30e7-4bab-8855-2a909fb4eb57\" (UID: \"01617010-30e7-4bab-8855-2a909fb4eb57\") "
Nov 03 21:15:17.943031 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.942929    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01617010-30e7-4bab-8855-2a909fb4eb57-kube-api-access-tt4cb\" (UniqueName: \"kubernetes.io/projected/01617010-30e7-4bab-8855-2a909fb4eb57-kube-api-access-tt4cb\") pod \"01617010-30e7-4bab-8855-2a909fb4eb57\" (UID: \"01617010-30e7-4bab-8855-2a909fb4eb57\") "
Nov 03 21:15:17.943031 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.942985    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e78536-6f08-4925-8493-e8958635e812-kube-api-access-klblz\" (UniqueName: \"kubernetes.io/projected/43e78536-6f08-4925-8493-e8958635e812-kube-api-access-klblz\") pod \"43e78536-6f08-4925-8493-e8958635e812\" (UID: \"43e78536-6f08-4925-8493-e8958635e812\") "
Nov 03 21:15:17.943267 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.943219    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e78536-6f08-4925-8493-e8958635e812-maas-api-service-ca" pod "43e78536-6f08-4925-8493-e8958635e812" (UID: "43e78536-6f08-4925-8493-e8958635e812"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:17.943267 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.943248    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01617010-30e7-4bab-8855-2a909fb4eb57-maas-api-service-ca" pod "01617010-30e7-4bab-8855-2a909fb4eb57" (UID: "01617010-30e7-4bab-8855-2a909fb4eb57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:17.943417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.943392    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e78536-6f08-4925-8493-e8958635e812-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.943417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.943413    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01617010-30e7-4bab-8855-2a909fb4eb57-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.945420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.945397    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e78536-6f08-4925-8493-e8958635e812-kube-api-access-klblz" pod "43e78536-6f08-4925-8493-e8958635e812" (UID: "43e78536-6f08-4925-8493-e8958635e812"). InnerVolumeSpecName "kube-api-access-klblz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:17.945527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.945428    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01617010-30e7-4bab-8855-2a909fb4eb57-kube-api-access-tt4cb" pod "01617010-30e7-4bab-8855-2a909fb4eb57" (UID: "01617010-30e7-4bab-8855-2a909fb4eb57"). InnerVolumeSpecName "kube-api-access-tt4cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:17.973903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:17.973866    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"]
Nov 03 21:15:18.044331 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.044298    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt4cb\" (UniqueName: \"kubernetes.io/projected/01617010-30e7-4bab-8855-2a909fb4eb57-kube-api-access-tt4cb\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:18.044331 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.044330    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klblz\" (UniqueName: \"kubernetes.io/projected/43e78536-6f08-4925-8493-e8958635e812-kube-api-access-klblz\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:18.110191 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.110153    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e1685f-1649-4108-b780-d02178c726e5" path="/var/lib/kubelet/pods/43e1685f-1649-4108-b780-d02178c726e5/volumes"
Nov 03 21:15:18.488339 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:18.488209    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:15:18.665956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.665919    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"
Nov 03 21:15:18.666475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.665949    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb" event={"ID":"43e78536-6f08-4925-8493-e8958635e812","Type":"ContainerDied","Data":"431ebe94b3f66c796fc0f425f1b73b6b179feb5228faa53ff4b5f8acd73eba15"}
Nov 03 21:15:18.666475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.666010    5196 scope.go:122] "RemoveContainer" containerID="721edef18953934eb100248fb2d77f7256bc18dc6d71dc67c3c6c2214b6f7cdc"
Nov 03 21:15:18.672859 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.672820    5196 generic.go:356] "Generic (PLEG): container finished" podID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerID="633ef35fa4371b1691b50fe9af7b7bffc56e997241bf12335e4d7d57382955ce" exitCode=60
Nov 03 21:15:18.672975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.672882    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" event={"ID":"45f49561-ad85-4e2e-b142-7fe299d71b61","Type":"ContainerDied","Data":"633ef35fa4371b1691b50fe9af7b7bffc56e997241bf12335e4d7d57382955ce"}
Nov 03 21:15:18.673404 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.673373    5196 scope.go:122] "RemoveContainer" containerID="633ef35fa4371b1691b50fe9af7b7bffc56e997241bf12335e4d7d57382955ce"
Nov 03 21:15:18.673712 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:18.673688    5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq_odh-ai-gateway-infra(45f49561-ad85-4e2e-b142-7fe299d71b61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61"
Nov 03 21:15:18.674924 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.674903    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm" event={"ID":"01617010-30e7-4bab-8855-2a909fb4eb57","Type":"ContainerDied","Data":"5eb25a2553a60229b798312a7487ddb7890c3fea1f79da07b41996dc5a125138"}
Nov 03 21:15:18.675026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.674933    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"
Nov 03 21:15:18.680770 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.680745    5196 scope.go:122] "RemoveContainer" containerID="0f4fd92818fffb986126ef5245f4038f34008a3a602fc1469e53441a9474693c"
Nov 03 21:15:18.698966 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.698944    5196 scope.go:122] "RemoveContainer" containerID="60479dbc5054264bfa552fd7ccc76e8f9f535fc7509a4dd8e73952d1211c1855"
Nov 03 21:15:18.840083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.840050    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:18.888359 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.888322    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"]
Nov 03 21:15:18.891975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.891939    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:18.892629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.892608    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-hvmzm"]
Nov 03 21:15:18.954409 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.954373    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78ed0bc-954a-4a1f-95f1-b3f25c404960-kube-api-access-xchz8\" (UniqueName: \"kubernetes.io/projected/d78ed0bc-954a-4a1f-95f1-b3f25c404960-kube-api-access-xchz8\") pod \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\" (UID: \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\") "
Nov 03 21:15:18.954620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.954467    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00856d79-dc98-421c-8d49-a74b92de46e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00856d79-dc98-421c-8d49-a74b92de46e8-maas-api-service-ca\") pod \"00856d79-dc98-421c-8d49-a74b92de46e8\" (UID: \"00856d79-dc98-421c-8d49-a74b92de46e8\") "
Nov 03 21:15:18.954620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.954547    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78ed0bc-954a-4a1f-95f1-b3f25c404960-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ed0bc-954a-4a1f-95f1-b3f25c404960-maas-api-service-ca\") pod \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\" (UID: \"d78ed0bc-954a-4a1f-95f1-b3f25c404960\") "
Nov 03 21:15:18.954620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.954565    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00856d79-dc98-421c-8d49-a74b92de46e8-kube-api-access-5555t\" (UniqueName: \"kubernetes.io/projected/00856d79-dc98-421c-8d49-a74b92de46e8-kube-api-access-5555t\") pod \"00856d79-dc98-421c-8d49-a74b92de46e8\" (UID: \"00856d79-dc98-421c-8d49-a74b92de46e8\") "
Nov 03 21:15:18.954910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.954883    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00856d79-dc98-421c-8d49-a74b92de46e8-maas-api-service-ca" pod "00856d79-dc98-421c-8d49-a74b92de46e8" (UID: "00856d79-dc98-421c-8d49-a74b92de46e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:18.954977 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.954900    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78ed0bc-954a-4a1f-95f1-b3f25c404960-maas-api-service-ca" pod "d78ed0bc-954a-4a1f-95f1-b3f25c404960" (UID: "d78ed0bc-954a-4a1f-95f1-b3f25c404960"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:18.956934 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.956912    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78ed0bc-954a-4a1f-95f1-b3f25c404960-kube-api-access-xchz8" pod "d78ed0bc-954a-4a1f-95f1-b3f25c404960" (UID: "d78ed0bc-954a-4a1f-95f1-b3f25c404960"). InnerVolumeSpecName "kube-api-access-xchz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:18.957006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:18.956945    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00856d79-dc98-421c-8d49-a74b92de46e8-kube-api-access-5555t" pod "00856d79-dc98-421c-8d49-a74b92de46e8" (UID: "00856d79-dc98-421c-8d49-a74b92de46e8"). InnerVolumeSpecName "kube-api-access-5555t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:19.055666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.055577    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xchz8\" (UniqueName: \"kubernetes.io/projected/d78ed0bc-954a-4a1f-95f1-b3f25c404960-kube-api-access-xchz8\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.055666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.055607    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00856d79-dc98-421c-8d49-a74b92de46e8-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.055666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.055621    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ed0bc-954a-4a1f-95f1-b3f25c404960-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.055666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.055630    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5555t\" (UniqueName: \"kubernetes.io/projected/00856d79-dc98-421c-8d49-a74b92de46e8-kube-api-access-5555t\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.099469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.099421    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"]
Nov 03 21:15:19.105657 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.105622    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-98d1d2-29895675-tjbrb"]
Nov 03 21:15:19.348328 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.348291    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"]
Nov 03 21:15:19.685231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.685140    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"
Nov 03 21:15:19.685231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.685156    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l" event={"ID":"d78ed0bc-954a-4a1f-95f1-b3f25c404960","Type":"ContainerDied","Data":"b91ce34c1b148f50413c00f3006da29b22c5bf93c2a4b068bff3e7aa9f6bf1a2"}
Nov 03 21:15:19.685231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.685229    5196 scope.go:122] "RemoveContainer" containerID="0b08f4d68624e5e2bfd7b15b2809bea92e2d4b2d8b120fdd24023a7a0e1d2892"
Nov 03 21:15:19.688215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.688193    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"
Nov 03 21:15:19.688215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.688202    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l" event={"ID":"00856d79-dc98-421c-8d49-a74b92de46e8","Type":"ContainerDied","Data":"82d8e81944e3a0a6dd88afa003e089de4a6ab0873be2fbdf69d6030cfe9ccbfa"}
Nov 03 21:15:19.698836 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.698810    5196 scope.go:122] "RemoveContainer" containerID="b77066ad8353d746f07adbde06e8e7c56a93fa23510e361d231478ee5a1e069f"
Nov 03 21:15:19.842025 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.841986    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"]
Nov 03 21:15:19.845933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.845898    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-562eb2-29895675-cc55l"]
Nov 03 21:15:19.850968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.850942    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:19.949208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.949125    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"]
Nov 03 21:15:19.952526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.952486    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-984cbc-29895675-jxz9l"]
Nov 03 21:15:19.964991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.964954    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f49561-ad85-4e2e-b142-7fe299d71b61-kube-api-access-x5rt7\" (UniqueName: \"kubernetes.io/projected/45f49561-ad85-4e2e-b142-7fe299d71b61-kube-api-access-x5rt7\") pod \"45f49561-ad85-4e2e-b142-7fe299d71b61\" (UID: \"45f49561-ad85-4e2e-b142-7fe299d71b61\") "
Nov 03 21:15:19.965179 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.965049    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f49561-ad85-4e2e-b142-7fe299d71b61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49561-ad85-4e2e-b142-7fe299d71b61-maas-api-service-ca\") pod \"45f49561-ad85-4e2e-b142-7fe299d71b61\" (UID: \"45f49561-ad85-4e2e-b142-7fe299d71b61\") "
Nov 03 21:15:19.965464 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.965439    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f49561-ad85-4e2e-b142-7fe299d71b61-maas-api-service-ca" pod "45f49561-ad85-4e2e-b142-7fe299d71b61" (UID: "45f49561-ad85-4e2e-b142-7fe299d71b61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:19.965565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.965547    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f49561-ad85-4e2e-b142-7fe299d71b61-maas-api-service-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.967436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:19.967415    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f49561-ad85-4e2e-b142-7fe299d71b61-kube-api-access-x5rt7" pod "45f49561-ad85-4e2e-b142-7fe299d71b61" (UID: "45f49561-ad85-4e2e-b142-7fe299d71b61"). InnerVolumeSpecName "kube-api-access-x5rt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:20.066380 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.066339    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5rt7\" (UniqueName: \"kubernetes.io/projected/45f49561-ad85-4e2e-b142-7fe299d71b61-kube-api-access-x5rt7\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.109967 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.109931    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" path="/var/lib/kubelet/pods/00856d79-dc98-421c-8d49-a74b92de46e8/volumes"
Nov 03 21:15:20.110434 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.110418    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" path="/var/lib/kubelet/pods/01617010-30e7-4bab-8855-2a909fb4eb57/volumes"
Nov 03 21:15:20.110873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.110857    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e78536-6f08-4925-8493-e8958635e812" path="/var/lib/kubelet/pods/43e78536-6f08-4925-8493-e8958635e812/volumes"
Nov 03 21:15:20.111573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.111526    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" path="/var/lib/kubelet/pods/d78ed0bc-954a-4a1f-95f1-b3f25c404960/volumes"
Nov 03 21:15:20.706723 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.706693    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"
Nov 03 21:15:20.707313 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.706719    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq" event={"ID":"45f49561-ad85-4e2e-b142-7fe299d71b61","Type":"ContainerDied","Data":"63b716aa78d8e4cf1dfa4de2097540c8bf0dba9391bf4beeeb67213f5c90be5a"}
Nov 03 21:15:20.707313 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.706794    5196 scope.go:122] "RemoveContainer" containerID="633ef35fa4371b1691b50fe9af7b7bffc56e997241bf12335e4d7d57382955ce"
Nov 03 21:15:20.863337 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.863302    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"]
Nov 03 21:15:20.909761 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:20.909727    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7c34ff-29895675-9gxtq"]
Nov 03 21:15:21.951552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:21.951512    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:22.108910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:22.108866    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" path="/var/lib/kubelet/pods/45f49561-ad85-4e2e-b142-7fe299d71b61/volumes"
Nov 03 21:15:23.121019 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:23.120979    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:23.272333 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:23.272292    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:24.235153 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.229182    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"]
Nov 03 21:15:24.257560 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.257510    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:24.786353 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786311    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"]
Nov 03 21:15:24.786904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786889    5196 cpu_manager.go:414] "containerMap: removing container" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:24.786965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786905    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:24.786965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786923    5196 cpu_manager.go:414] "containerMap: removing container" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:24.786965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786929    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:24.786965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786940    5196 cpu_manager.go:414] "containerMap: removing container" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:24.786965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786947    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:24.786965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786961    5196 cpu_manager.go:414] "containerMap: removing container" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:24.786965 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786966    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786972    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786976    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786987    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786991    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.786997    5196 cpu_manager.go:414] "containerMap: removing container" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787002    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787009    5196 cpu_manager.go:414] "containerMap: removing container" podUID="7789434b-0951-4f26-b333-ad9fded40ecc" containerName="revoke-keys"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787013    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7789434b-0951-4f26-b333-ad9fded40ecc" containerName="revoke-keys"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787021    5196 cpu_manager.go:414] "containerMap: removing container" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787025    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787035    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787040    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787047    5196 cpu_manager.go:414] "containerMap: removing container" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787053    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787061    5196 cpu_manager.go:414] "containerMap: removing container" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787065    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787076    5196 cpu_manager.go:414] "containerMap: removing container" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787080    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787104    5196 cpu_manager.go:414] "containerMap: removing container" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787111    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787121    5196 cpu_manager.go:414] "containerMap: removing container" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787127    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787246    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787255    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787261    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787266    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7789434b-0951-4f26-b333-ad9fded40ecc" containerName="revoke-keys"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787274    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787283    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787287    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787292    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787298    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787303    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787308    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787314    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787320    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e78536-6f08-4925-8493-e8958635e812" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787327    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:24.787344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.787333    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:24.792278 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.792254    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:24.841505 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.841471    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:15:24.841505 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.841469    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:15:24.844024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.843996    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"]
Nov 03 21:15:24.926928 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.926888    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:24.926928 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.926929    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gmsf\" (UniqueName: \"kubernetes.io/projected/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-kube-api-access-9gmsf\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:24.927266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:24.926970    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:25.028860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.028816    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:25.029047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.028986    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:25.029047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.029008    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gmsf\" (UniqueName: \"kubernetes.io/projected/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-kube-api-access-9gmsf\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:25.034254 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.030276    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:25.041153 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.041038    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:25.083055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.083019    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gmsf\" (UniqueName: \"kubernetes.io/projected/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-kube-api-access-9gmsf\") pod \"maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:25.108030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.107993    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:25.488234 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.488193    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"]
Nov 03 21:15:25.494186 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:25.494154    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9245c0ad_5497_4b78_b3f6_7b91a7ab6fb0.slice/crio-ee171d746a145ed52844a8a7abbdb7e4dd16f40625f09bed4c2df5c3b9177b10 WatchSource:0}: Error finding container ee171d746a145ed52844a8a7abbdb7e4dd16f40625f09bed4c2df5c3b9177b10: Status 404 returned error can't find the container with id ee171d746a145ed52844a8a7abbdb7e4dd16f40625f09bed4c2df5c3b9177b10
Nov 03 21:15:25.763301 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.763259    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6" event={"ID":"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0","Type":"ContainerStarted","Data":"2196aee6fc13721ac4176bcb169156acc8aef02c786f39545e6790b9d7aec191"}
Nov 03 21:15:25.763486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.763330    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6" event={"ID":"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0","Type":"ContainerStarted","Data":"ee171d746a145ed52844a8a7abbdb7e4dd16f40625f09bed4c2df5c3b9177b10"}
Nov 03 21:15:25.873490 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:25.873418    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6" podStartSLOduration=1.8734002749999998 podStartE2EDuration="1.873400275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:25.870029768 +0000 UTC m=+2765.873988525" watchObservedRunningTime="2026-11-03 21:15:25.873400275 +0000 UTC m=+2765.877359000"
Nov 03 21:15:26.773317 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:26.773266    5196 generic.go:356] "Generic (PLEG): container finished" podID="9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0" containerID="2196aee6fc13721ac4176bcb169156acc8aef02c786f39545e6790b9d7aec191" exitCode=0
Nov 03 21:15:26.773734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:26.773352    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6" event={"ID":"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0","Type":"ContainerDied","Data":"2196aee6fc13721ac4176bcb169156acc8aef02c786f39545e6790b9d7aec191"}
Nov 03 21:15:26.912991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:26.912949    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:27.398731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:27.398697    5196 scope.go:122] "RemoveContainer" containerID="5c06e89f12be429e7d91808f865e585d4a1b97527104611bf090713c1473ca09"
Nov 03 21:15:27.410521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:27.410495    5196 scope.go:122] "RemoveContainer" containerID="ce55b61852b879e8479f032ec646132dae186a3c8b402f5265093f706de963bf"
Nov 03 21:15:27.421027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:27.421006    5196 scope.go:122] "RemoveContainer" containerID="7810d5208786999cdd53136e2a55fe50557167fe37c7338a1d06d64043a61028"
Nov 03 21:15:27.435534 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:27.435506    5196 scope.go:122] "RemoveContainer" containerID="e46ce919b76e9cf3c567f0752106406c13cd94c5180fd63a12c1055056660ffa"
Nov 03 21:15:27.446609 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:27.446589    5196 scope.go:122] "RemoveContainer" containerID="04a485c41af00093f2c967f125e3d4f6a2e5ebe6e395f79e6de42674f669eb8a"
Nov 03 21:15:27.458914 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:27.458891    5196 scope.go:122] "RemoveContainer" containerID="fc0a3667b4d40d26297be480944e9203cf1be5d9bcd8b511d71d9567be39d960"
Nov 03 21:15:27.960174 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:27.960141    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:28.068224 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.068193    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-xks-ca\") pod \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") "
Nov 03 21:15:28.068415 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.068368    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-openshift-ca\") pod \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") "
Nov 03 21:15:28.068504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.068472    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-kube-api-access-9gmsf\" (UniqueName: \"kubernetes.io/projected/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-kube-api-access-9gmsf\") pod \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\" (UID: \"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0\") "
Nov 03 21:15:28.069239 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.069208    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-openshift-ca" pod "9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0" (UID: "9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:28.071662 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.071618    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-kube-api-access-9gmsf" pod "9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0" (UID: "9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0"). InnerVolumeSpecName "kube-api-access-9gmsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:28.072621 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.072598    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-xks-ca" pod "9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0" (UID: "9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:28.170732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.170689    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gmsf\" (UniqueName: \"kubernetes.io/projected/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-kube-api-access-9gmsf\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:28.170732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.170732    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:28.170732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.170744    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:28.791195 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.791155    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6" event={"ID":"9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0","Type":"ContainerDied","Data":"ee171d746a145ed52844a8a7abbdb7e4dd16f40625f09bed4c2df5c3b9177b10"}
Nov 03 21:15:28.791195 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.791185    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"
Nov 03 21:15:28.791195 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:28.791204    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee171d746a145ed52844a8a7abbdb7e4dd16f40625f09bed4c2df5c3b9177b10"
Nov 03 21:15:32.095248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:32.095198    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:36.652130 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:36.652070    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:36.806848 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:36.806806    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:37.066127 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:37.066002    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:39.184299 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.184261    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:15:39.255898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.255848    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k" podUID="8fd3fbdc-5419-4b73-84b9-375fc52ace96" containerName="main" containerID="cri-o://b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392" gracePeriod=45
Nov 03 21:15:39.536925 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.536898    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:15:39.599418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599382    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-model-cache\") pod \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") "
Nov 03 21:15:39.599614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599481    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tmp-dir\") pod \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") "
Nov 03 21:15:39.599614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599531    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-home\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-home\") pod \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") "
Nov 03 21:15:39.599614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599553    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tls-certs\") pod \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") "
Nov 03 21:15:39.599614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599572    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd3fbdc-5419-4b73-84b9-375fc52ace96-kube-api-access-74pzd\" (UniqueName: \"kubernetes.io/projected/8fd3fbdc-5419-4b73-84b9-375fc52ace96-kube-api-access-74pzd\") pod \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") "
Nov 03 21:15:39.599614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599609    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-dshm\") pod \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\" (UID: \"8fd3fbdc-5419-4b73-84b9-375fc52ace96\") "
Nov 03 21:15:39.599888 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599702    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-model-cache" pod "8fd3fbdc-5419-4b73-84b9-375fc52ace96" (UID: "8fd3fbdc-5419-4b73-84b9-375fc52ace96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:39.599888 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599800    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-home" pod "8fd3fbdc-5419-4b73-84b9-375fc52ace96" (UID: "8fd3fbdc-5419-4b73-84b9-375fc52ace96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:39.599888 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.599809    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tmp-dir" pod "8fd3fbdc-5419-4b73-84b9-375fc52ace96" (UID: "8fd3fbdc-5419-4b73-84b9-375fc52ace96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:39.600068 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.600052    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:39.600137 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.600070    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:39.600137 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.600080    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:39.601945 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.601920    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-dshm" pod "8fd3fbdc-5419-4b73-84b9-375fc52ace96" (UID: "8fd3fbdc-5419-4b73-84b9-375fc52ace96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:39.602058 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.602027    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd3fbdc-5419-4b73-84b9-375fc52ace96-kube-api-access-74pzd" pod "8fd3fbdc-5419-4b73-84b9-375fc52ace96" (UID: "8fd3fbdc-5419-4b73-84b9-375fc52ace96"). InnerVolumeSpecName "kube-api-access-74pzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:39.602157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.602049    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tls-certs" pod "8fd3fbdc-5419-4b73-84b9-375fc52ace96" (UID: "8fd3fbdc-5419-4b73-84b9-375fc52ace96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:39.701454 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.701367    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fd3fbdc-5419-4b73-84b9-375fc52ace96-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:39.701454 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.701401    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74pzd\" (UniqueName: \"kubernetes.io/projected/8fd3fbdc-5419-4b73-84b9-375fc52ace96-kube-api-access-74pzd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:39.701454 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.701412    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fbdc-5419-4b73-84b9-375fc52ace96-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:39.881993 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.881962    5196 generic.go:356] "Generic (PLEG): container finished" podID="8fd3fbdc-5419-4b73-84b9-375fc52ace96" containerID="b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392" exitCode=0
Nov 03 21:15:39.882192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.882009    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k" event={"ID":"8fd3fbdc-5419-4b73-84b9-375fc52ace96","Type":"ContainerDied","Data":"b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392"}
Nov 03 21:15:39.882192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.882023    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"
Nov 03 21:15:39.882192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.882041    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k" event={"ID":"8fd3fbdc-5419-4b73-84b9-375fc52ace96","Type":"ContainerDied","Data":"63907df8b796e0f53f3fff71e574b855a76b68b7a6124e652887132616aeac67"}
Nov 03 21:15:39.882192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.882057    5196 scope.go:122] "RemoveContainer" containerID="b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392"
Nov 03 21:15:39.893884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.893859    5196 scope.go:122] "RemoveContainer" containerID="b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392"
Nov 03 21:15:39.894226 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:39.894207    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392\": container with ID starting with b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392 not found: ID does not exist" containerID="b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392"
Nov 03 21:15:39.894302 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:39.894235    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392"} err="failed to get container status \"b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392\": rpc error: code = NotFound desc = could not find container \"b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392\": container with ID starting with b402da1249417f9aeb6205a285c603b63d65367899b8a35a8aa5d1438b246392 not found: ID does not exist"
Nov 03 21:15:40.038775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:40.038683    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"]
Nov 03 21:15:40.045330 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:40.045301    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-98d1d2/test-model-98d1d2-kserve-7c79bfc655-4qz4k"]
Nov 03 21:15:40.117619 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:40.117582    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd3fbdc-5419-4b73-84b9-375fc52ace96" path="/var/lib/kubelet/pods/8fd3fbdc-5419-4b73-84b9-375fc52ace96/volumes"
Nov 03 21:15:41.195718 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:41.195678    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:42.760247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:42.760209    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:42.979746 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:42.979686    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:43.129300 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:43.129261    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:43.362583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:43.362533    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"]
Nov 03 21:15:43.367645 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:43.367608    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c14420-fq8f8"]
Nov 03 21:15:43.790728 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:43.790690    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:44.109666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:44.109633    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7789434b-0951-4f26-b333-ad9fded40ecc" path="/var/lib/kubelet/pods/7789434b-0951-4f26-b333-ad9fded40ecc/volumes"
Nov 03 21:15:47.407741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:47.407702    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:48.485383 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:15:48.485336    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:15:48.940718 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:48.940670    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:49.166925 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:49.166884    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:49.316053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:49.315961    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:49.420400 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:49.420365    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"]
Nov 03 21:15:49.467264 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:49.467232    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c34ff-gchd6"]
Nov 03 21:15:49.815105 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:49.815060    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:50.013722 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:50.013689    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:50.111187 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:50.111149    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0" path="/var/lib/kubelet/pods/9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0/volumes"
Nov 03 21:15:50.319904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:50.319791    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:51.031718 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.031672    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"]
Nov 03 21:15:51.032341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032324    5196 cpu_manager.go:414] "containerMap: removing container" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:51.032406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032342    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:51.032406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032355    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0" containerName="revoke-keys"
Nov 03 21:15:51.032406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032360    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0" containerName="revoke-keys"
Nov 03 21:15:51.032406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032383    5196 cpu_manager.go:414] "containerMap: removing container" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:51.032406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032388    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:51.032406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032406    5196 cpu_manager.go:414] "containerMap: removing container" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032411    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00856d79-dc98-421c-8d49-a74b92de46e8" containerName="cleanup"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032417    5196 cpu_manager.go:414] "containerMap: removing container" podUID="8fd3fbdc-5419-4b73-84b9-375fc52ace96" containerName="main"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032422    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd3fbdc-5419-4b73-84b9-375fc52ace96" containerName="main"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032437    5196 cpu_manager.go:414] "containerMap: removing container" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032442    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032521    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f49561-ad85-4e2e-b142-7fe299d71b61" containerName="cleanup"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032531    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9245c0ad-5497-4b78-b3f6-7b91a7ab6fb0" containerName="revoke-keys"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032553    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78ed0bc-954a-4a1f-95f1-b3f25c404960" containerName="cleanup"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032560    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e1685f-1649-4108-b780-d02178c726e5" containerName="cleanup"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032567    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="01617010-30e7-4bab-8855-2a909fb4eb57" containerName="cleanup"
Nov 03 21:15:51.032598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.032574    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd3fbdc-5419-4b73-84b9-375fc52ace96" containerName="main"
Nov 03 21:15:51.037306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.037266    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.081429 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.081389    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"]
Nov 03 21:15:51.118513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.118472    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.118745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.118656    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwsqw\" (UniqueName: \"kubernetes.io/projected/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-kube-api-access-lwsqw\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.118745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.118724    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.123694 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.123669    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:15:51.123794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.123736    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:15:51.220057 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.220015    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.220289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.220154    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwsqw\" (UniqueName: \"kubernetes.io/projected/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-kube-api-access-lwsqw\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.220289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.220210    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.220900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.220873    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.222713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.222684    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.274199 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.274163    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwsqw\" (UniqueName: \"kubernetes.io/projected/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-kube-api-access-lwsqw\") pod \"maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.352265 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.352230    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:51.488983 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.488947    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:51.734564 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.734473    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"]
Nov 03 21:15:51.741150 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:51.741111    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4efb181_5a0b_4dd5_9fed_26868ca7cb57.slice/crio-cf41cb67af7fde3f9a0756be8fe01d49dac530c1f7d5322d6ca9cdd89434bf26 WatchSource:0}: Error finding container cf41cb67af7fde3f9a0756be8fe01d49dac530c1f7d5322d6ca9cdd89434bf26: Status 404 returned error can't find the container with id cf41cb67af7fde3f9a0756be8fe01d49dac530c1f7d5322d6ca9cdd89434bf26
Nov 03 21:15:51.993383 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:51.993295    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf" event={"ID":"a4efb181-5a0b-4dd5-9fed-26868ca7cb57","Type":"ContainerStarted","Data":"cf41cb67af7fde3f9a0756be8fe01d49dac530c1f7d5322d6ca9cdd89434bf26"}
Nov 03 21:15:53.002823 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:53.002728    5196 generic.go:356] "Generic (PLEG): container finished" podID="a4efb181-5a0b-4dd5-9fed-26868ca7cb57" containerID="e341433f0d8c053114bdb110df5f7307eb3ce7aa43ea14bba80e6a3d98d222b0" exitCode=0
Nov 03 21:15:53.003309 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:53.002824    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf" event={"ID":"a4efb181-5a0b-4dd5-9fed-26868ca7cb57","Type":"ContainerDied","Data":"e341433f0d8c053114bdb110df5f7307eb3ce7aa43ea14bba80e6a3d98d222b0"}
Nov 03 21:15:54.178531 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.178501    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:54.262810 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.262775    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-kube-api-access-lwsqw\" (UniqueName: \"kubernetes.io/projected/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-kube-api-access-lwsqw\") pod \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") "
Nov 03 21:15:54.262991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.262917    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-xks-ca\") pod \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") "
Nov 03 21:15:54.262991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.262955    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-openshift-ca\") pod \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\" (UID: \"a4efb181-5a0b-4dd5-9fed-26868ca7cb57\") "
Nov 03 21:15:54.263418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.263392    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-openshift-ca" pod "a4efb181-5a0b-4dd5-9fed-26868ca7cb57" (UID: "a4efb181-5a0b-4dd5-9fed-26868ca7cb57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:54.263699 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.263503    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:54.265155 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.265130    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-kube-api-access-lwsqw" pod "a4efb181-5a0b-4dd5-9fed-26868ca7cb57" (UID: "a4efb181-5a0b-4dd5-9fed-26868ca7cb57"). InnerVolumeSpecName "kube-api-access-lwsqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:54.265266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.265246    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-xks-ca" pod "a4efb181-5a0b-4dd5-9fed-26868ca7cb57" (UID: "a4efb181-5a0b-4dd5-9fed-26868ca7cb57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:54.364384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.364339    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwsqw\" (UniqueName: \"kubernetes.io/projected/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-kube-api-access-lwsqw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:54.364384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.364378    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4efb181-5a0b-4dd5-9fed-26868ca7cb57-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:54.447531 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.447489    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:54.815501 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:54.815462    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:55.022695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:55.022650    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf" event={"ID":"a4efb181-5a0b-4dd5-9fed-26868ca7cb57","Type":"ContainerDied","Data":"cf41cb67af7fde3f9a0756be8fe01d49dac530c1f7d5322d6ca9cdd89434bf26"}
Nov 03 21:15:55.022880 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:55.022717    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf41cb67af7fde3f9a0756be8fe01d49dac530c1f7d5322d6ca9cdd89434bf26"
Nov 03 21:15:55.022880 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:55.022665    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"
Nov 03 21:15:55.546420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:55.546355    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:55.749713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:55.749669    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:55.914370 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:55.914330    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:56.684809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.684776    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"]
Nov 03 21:15:56.685480 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.685459    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a4efb181-5a0b-4dd5-9fed-26868ca7cb57" containerName="revoke-keys"
Nov 03 21:15:56.685480 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.685479    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4efb181-5a0b-4dd5-9fed-26868ca7cb57" containerName="revoke-keys"
Nov 03 21:15:56.685631 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.685619    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4efb181-5a0b-4dd5-9fed-26868ca7cb57" containerName="revoke-keys"
Nov 03 21:15:56.689491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.689467    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.741837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.741800    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"]
Nov 03 21:15:56.783933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.783901    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:15:56.784160 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.784023    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:15:56.795571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.795536    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.795775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.795604    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.795834 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.795806    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxvg\" (UniqueName: \"kubernetes.io/projected/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-kube-api-access-jrxvg\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.896935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.896899    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxvg\" (UniqueName: \"kubernetes.io/projected/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-kube-api-access-jrxvg\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.897167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.896988    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.897167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.897025    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.897748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.897720    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.899756 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.899725    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:56.951337 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:56.951262    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxvg\" (UniqueName: \"kubernetes.io/projected/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-kube-api-access-jrxvg\") pod \"maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:57.005420 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:57.005383    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:15:57.386182 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:57.386147    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"]
Nov 03 21:15:57.392235 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:15:57.392202    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f472e3c_ab4c_4dcf_b2d7_d150f0bd51ca.slice/crio-f8c86e7b7dc7f2d24150d2fe0eb60d648dc68a14884487effd02f0f8aaab4b2b WatchSource:0}: Error finding container f8c86e7b7dc7f2d24150d2fe0eb60d648dc68a14884487effd02f0f8aaab4b2b: Status 404 returned error can't find the container with id f8c86e7b7dc7f2d24150d2fe0eb60d648dc68a14884487effd02f0f8aaab4b2b
Nov 03 21:15:58.045971 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.045881    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"]
Nov 03 21:15:58.050656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.050624    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448" event={"ID":"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca","Type":"ContainerStarted","Data":"5d81673e6f1c2a6b2547823e65170366106982af84c8b8bb720d95f3f8951fea"}
Nov 03 21:15:58.050807 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.050678    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448" event={"ID":"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca","Type":"ContainerStarted","Data":"f8c86e7b7dc7f2d24150d2fe0eb60d648dc68a14884487effd02f0f8aaab4b2b"}
Nov 03 21:15:58.091658 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.091625    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"]
Nov 03 21:15:58.094736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.094724    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175f5c05-6f28p"]
Nov 03 21:15:58.105891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.100399    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a87926-b4j2f"]
Nov 03 21:15:58.114559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.114520    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51807c06-0f48-443f-ae03-43040b440a23" path="/var/lib/kubelet/pods/51807c06-0f48-443f-ae03-43040b440a23/volumes"
Nov 03 21:15:58.115021 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.115000    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91547460-ae34-4873-81d3-c18bc132a837" path="/var/lib/kubelet/pods/91547460-ae34-4873-81d3-c18bc132a837/volumes"
Nov 03 21:15:58.169601 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:58.169545    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448" podStartSLOduration=2.169529293 podStartE2EDuration="2.169529293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:58.16585626 +0000 UTC m=+2798.169815006" watchObservedRunningTime="2026-11-03 21:15:58.169529293 +0000 UTC m=+2798.173488017"
Nov 03 21:15:59.059852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:59.059818    5196 generic.go:356] "Generic (PLEG): container finished" podID="5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca" containerID="5d81673e6f1c2a6b2547823e65170366106982af84c8b8bb720d95f3f8951fea" exitCode=0
Nov 03 21:15:59.060306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:59.059901    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448" event={"ID":"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca","Type":"ContainerDied","Data":"5d81673e6f1c2a6b2547823e65170366106982af84c8b8bb720d95f3f8951fea"}
Nov 03 21:15:59.757460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:15:59.757418    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:00.224073 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.224042    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:16:00.342296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.342212    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-openshift-ca\") pod \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") "
Nov 03 21:16:00.342296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.342270    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-xks-ca\") pod \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") "
Nov 03 21:16:00.342541 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.342382    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-kube-api-access-jrxvg\" (UniqueName: \"kubernetes.io/projected/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-kube-api-access-jrxvg\") pod \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\" (UID: \"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca\") "
Nov 03 21:16:00.342681 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.342655    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-openshift-ca" pod "5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca" (UID: "5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:00.342854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.342833    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:00.344905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.344876    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-kube-api-access-jrxvg" pod "5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca" (UID: "5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca"). InnerVolumeSpecName "kube-api-access-jrxvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:00.345071 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.345055    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-xks-ca" pod "5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca" (UID: "5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:00.444451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.444417    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:00.444451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.444447    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrxvg\" (UniqueName: \"kubernetes.io/projected/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca-kube-api-access-jrxvg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:00.780612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:00.780522    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:16:01.076853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:01.076824    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"
Nov 03 21:16:01.077046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:01.076864    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448" event={"ID":"5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca","Type":"ContainerDied","Data":"f8c86e7b7dc7f2d24150d2fe0eb60d648dc68a14884487effd02f0f8aaab4b2b"}
Nov 03 21:16:01.077046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:01.076919    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8c86e7b7dc7f2d24150d2fe0eb60d648dc68a14884487effd02f0f8aaab4b2b"
Nov 03 21:16:01.866896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:01.866862    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"]
Nov 03 21:16:01.867470 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:01.867060    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy" containerID="cri-o://9c756a866f02c74d2a08f2b3ce0f6192d4793230dbfcbbf2992efecf561bb2e2" gracePeriod=30
Nov 03 21:16:02.512198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:02.512158    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"]
Nov 03 21:16:02.620572 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:02.620531    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"]
Nov 03 21:16:02.688957 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:02.688915    5196 patch_prober.go:29] interesting pod/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:16:02.689259 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:02.688981    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:16:02.895243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:02.895208    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:03.096615 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:03.096568    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:04.669842 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:04.669809    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:07.123927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.123894    5196 generic.go:356] "Generic (PLEG): container finished" podID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerID="9c756a866f02c74d2a08f2b3ce0f6192d4793230dbfcbbf2992efecf561bb2e2" exitCode=0
Nov 03 21:16:07.124322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.123947    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" event={"ID":"edee909c-37c4-4418-aabf-6ffc3cfe4267","Type":"ContainerDied","Data":"9c756a866f02c74d2a08f2b3ce0f6192d4793230dbfcbbf2992efecf561bb2e2"}
Nov 03 21:16:07.124322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.123986    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc" event={"ID":"edee909c-37c4-4418-aabf-6ffc3cfe4267","Type":"ContainerDied","Data":"cc6daa93925f6e9f71badee0a69ef8c844b41c217c98a6332000c0f775fc963c"}
Nov 03 21:16:07.124322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.124003    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc6daa93925f6e9f71badee0a69ef8c844b41c217c98a6332000c0f775fc963c"
Nov 03 21:16:07.135243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.135176    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 21:16:07.221747 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.221714    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-podinfo\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.221933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.221776    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-token\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-token\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.221933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.221818    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-envoy\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.221933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.221846    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edee909c-37c4-4418-aabf-6ffc3cfe4267-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee909c-37c4-4418-aabf-6ffc3cfe4267-istiod-ca-cert\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.221933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.221872    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-certs\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.221933 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.221923    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-socket\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.222271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.221966    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-data\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.222271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222039    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-kube-api-access-7scmn\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-kube-api-access-7scmn\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.222271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222141    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-credential-socket\") pod \"edee909c-37c4-4418-aabf-6ffc3cfe4267\" (UID: \"edee909c-37c4-4418-aabf-6ffc3cfe4267\") "
Nov 03 21:16:07.222271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222254    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-socket" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.222509 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222273    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-certs" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.222509 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222356    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-data" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.222509 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222378    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edee909c-37c4-4418-aabf-6ffc3cfe4267-istiod-ca-cert" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:07.222748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222554    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-credential-socket" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.222748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222704    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee909c-37c4-4418-aabf-6ffc3cfe4267-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.222748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222724    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.222748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222743    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.222895 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222759    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.222895 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.222775    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.224348 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.224315    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-podinfo" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:16:07.224657 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.224630    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-envoy" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.224832 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.224811    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-token" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:07.224876 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.224836    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-kube-api-access-7scmn" pod "edee909c-37c4-4418-aabf-6ffc3cfe4267" (UID: "edee909c-37c4-4418-aabf-6ffc3cfe4267"). InnerVolumeSpecName "kube-api-access-7scmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:07.324266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.324215    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.324266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.324259    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.324266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.324273    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee909c-37c4-4418-aabf-6ffc3cfe4267-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.324601 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.324287    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7scmn\" (UniqueName: \"kubernetes.io/projected/edee909c-37c4-4418-aabf-6ffc3cfe4267-kube-api-access-7scmn\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.512660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.512535    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c" podUID="f681aac0-0c6c-4dec-b8ea-9b837683c320" containerName="payload-processing" containerID="cri-o://cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10" gracePeriod=25
Nov 03 21:16:07.621898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:07.621828    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx" podUID="898fc4bb-d514-46e7-98a0-f457940b468f" containerName="payload-pre-processing" containerID="cri-o://70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b" gracePeriod=25
Nov 03 21:16:08.131189 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:08.131159    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"
Nov 03 21:16:08.334166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:08.334130    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"]
Nov 03 21:16:08.381032 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:08.380941    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fe2825-openshift-default-864db9667f-cxvnc"]
Nov 03 21:16:08.553582 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:08.553544    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:08.696677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:08.696585    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:09.378812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:09.378775    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:10.111079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:10.111042    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" path="/var/lib/kubelet/pods/edee909c-37c4-4418-aabf-6ffc3cfe4267/volumes"
Nov 03 21:16:12.333775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:12.333741    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:12.490040 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:12.490003    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:14.176321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:14.176281    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:14.888082 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:14.888033    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx" podUID="898fc4bb-d514-46e7-98a0-f457940b468f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:16:14.888082 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:14.888062    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c" podUID="f681aac0-0c6c-4dec-b8ea-9b837683c320" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:16:18.295323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:18.295289    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:18.486131 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:16:18.486071    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:16:19.420347 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:19.420308    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:19.791000 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:19.790913    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:20.154869 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:20.154828    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:20.268010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:20.267969    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"]
Nov 03 21:16:20.312974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:20.312913    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-562eb2-j6lgf"]
Nov 03 21:16:20.691079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:20.691040    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:21.005077 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:21.004988    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:21.262383 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:21.262297    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:21.412300 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:21.412265    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:21.563281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:21.563194    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"]
Nov 03 21:16:21.609985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:21.609950    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-98d1d2-7c448"]
Nov 03 21:16:22.027466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:22.027433    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:22.109387 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:22.109352    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca" path="/var/lib/kubelet/pods/5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca/volumes"
Nov 03 21:16:22.109812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:22.109799    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4efb181-5a0b-4dd5-9fed-26868ca7cb57" path="/var/lib/kubelet/pods/a4efb181-5a0b-4dd5-9fed-26868ca7cb57/volumes"
Nov 03 21:16:24.887484 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:24.887447    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c" podUID="f681aac0-0c6c-4dec-b8ea-9b837683c320" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:16:24.887927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:24.887484    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx" podUID="898fc4bb-d514-46e7-98a0-f457940b468f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:16:26.489432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:26.489397    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:26.638815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:26.638774    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:26.823840 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:26.823801    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:27.559161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.559076    5196 scope.go:122] "RemoveContainer" containerID="1a366e42b70e46de91a0c5ddc3220fdf7e65f40bf890e2decad0a381b0181006"
Nov 03 21:16:27.627939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.627835    5196 scope.go:122] "RemoveContainer" containerID="9c756a866f02c74d2a08f2b3ce0f6192d4793230dbfcbbf2992efecf561bb2e2"
Nov 03 21:16:27.650684 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.650654    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"]
Nov 03 21:16:27.651360 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.651340    5196 cpu_manager.go:414] "containerMap: removing container" podUID="5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca" containerName="revoke-keys"
Nov 03 21:16:27.651360 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.651360    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca" containerName="revoke-keys"
Nov 03 21:16:27.651492 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.651375    5196 cpu_manager.go:414] "containerMap: removing container" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy"
Nov 03 21:16:27.651492 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.651380    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy"
Nov 03 21:16:27.651492 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.651484    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f472e3c-ab4c-4dcf-b2d7-d150f0bd51ca" containerName="revoke-keys"
Nov 03 21:16:27.651492 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.651493    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="edee909c-37c4-4418-aabf-6ffc3cfe4267" containerName="istio-proxy"
Nov 03 21:16:27.654701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.654679    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.692367 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.692326    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:16:27.699713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.699688    5196 scope.go:122] "RemoveContainer" containerID="7277dda51a43854e7191d876fe0365d675bc231d123c24bb56c26ae91054e0d0"
Nov 03 21:16:27.721501 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.721401    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"]
Nov 03 21:16:27.736645 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.736594    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:16:27.737660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.737637    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:16:27.744000 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.743960    5196 scope.go:122] "RemoveContainer" containerID="8bc642887d65b27c49ea06060a0c0e44c99919f73205e6466fab85c4a236c3b8"
Nov 03 21:16:27.750741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.750712    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.750893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.750852    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.750974 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.750908    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkx6\" (UniqueName: \"kubernetes.io/projected/aa7b4055-35a6-45a0-8de9-5367671311ab-kube-api-access-pjkx6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.759779 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.759753    5196 scope.go:122] "RemoveContainer" containerID="b2f2c292071aebfdcd8c53ab1ffb59be6e7c0ce6862e5e76179bed50378f7f02"
Nov 03 21:16:27.802833 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.802813    5196 scope.go:122] "RemoveContainer" containerID="73c86e95c42132fe67fff5c889ef38296e0d2d830ce7ee7693a76e57d8792d29"
Nov 03 21:16:27.804558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.804517    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:16:27.815050 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.815032    5196 scope.go:122] "RemoveContainer" containerID="90f74c5d1c57653a4650f66294db0cc9c7c92cd6482a2d5f62f71700b71a9d3a"
Nov 03 21:16:27.851831 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.851796    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f681aac0-0c6c-4dec-b8ea-9b837683c320-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681aac0-0c6c-4dec-b8ea-9b837683c320-plugins-config-volume\") pod \"f681aac0-0c6c-4dec-b8ea-9b837683c320\" (UID: \"f681aac0-0c6c-4dec-b8ea-9b837683c320\") "
Nov 03 21:16:27.851975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.851942    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f681aac0-0c6c-4dec-b8ea-9b837683c320-kube-api-access-wnxq9\" (UniqueName: \"kubernetes.io/projected/f681aac0-0c6c-4dec-b8ea-9b837683c320-kube-api-access-wnxq9\") pod \"f681aac0-0c6c-4dec-b8ea-9b837683c320\" (UID: \"f681aac0-0c6c-4dec-b8ea-9b837683c320\") "
Nov 03 21:16:27.852381 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.852349    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.852773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.852745    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f681aac0-0c6c-4dec-b8ea-9b837683c320-plugins-config-volume" pod "f681aac0-0c6c-4dec-b8ea-9b837683c320" (UID: "f681aac0-0c6c-4dec-b8ea-9b837683c320"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:27.852852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.852800    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkx6\" (UniqueName: \"kubernetes.io/projected/aa7b4055-35a6-45a0-8de9-5367671311ab-kube-api-access-pjkx6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.852918 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.852876    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.852992 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.852977    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681aac0-0c6c-4dec-b8ea-9b837683c320-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:27.853227 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.853202    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.856377 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.856351    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f681aac0-0c6c-4dec-b8ea-9b837683c320-kube-api-access-wnxq9" pod "f681aac0-0c6c-4dec-b8ea-9b837683c320" (UID: "f681aac0-0c6c-4dec-b8ea-9b837683c320"). InnerVolumeSpecName "kube-api-access-wnxq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:27.859758 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.859732    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.912538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.912514    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkx6\" (UniqueName: \"kubernetes.io/projected/aa7b4055-35a6-45a0-8de9-5367671311ab-kube-api-access-pjkx6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:27.913570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.913454    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:16:27.953989 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.953951    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898fc4bb-d514-46e7-98a0-f457940b468f-kube-api-access-5b4jq\" (UniqueName: \"kubernetes.io/projected/898fc4bb-d514-46e7-98a0-f457940b468f-kube-api-access-5b4jq\") pod \"898fc4bb-d514-46e7-98a0-f457940b468f\" (UID: \"898fc4bb-d514-46e7-98a0-f457940b468f\") "
Nov 03 21:16:27.954188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.954025    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898fc4bb-d514-46e7-98a0-f457940b468f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fc4bb-d514-46e7-98a0-f457940b468f-plugins-config-volume\") pod \"898fc4bb-d514-46e7-98a0-f457940b468f\" (UID: \"898fc4bb-d514-46e7-98a0-f457940b468f\") "
Nov 03 21:16:27.954437 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.954422    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnxq9\" (UniqueName: \"kubernetes.io/projected/f681aac0-0c6c-4dec-b8ea-9b837683c320-kube-api-access-wnxq9\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:27.954544 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.954490    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898fc4bb-d514-46e7-98a0-f457940b468f-plugins-config-volume" pod "898fc4bb-d514-46e7-98a0-f457940b468f" (UID: "898fc4bb-d514-46e7-98a0-f457940b468f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:27.956445 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:27.956423    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898fc4bb-d514-46e7-98a0-f457940b468f-kube-api-access-5b4jq" pod "898fc4bb-d514-46e7-98a0-f457940b468f" (UID: "898fc4bb-d514-46e7-98a0-f457940b468f"). InnerVolumeSpecName "kube-api-access-5b4jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:28.036176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.034602    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:28.056205 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.056127    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4jq\" (UniqueName: \"kubernetes.io/projected/898fc4bb-d514-46e7-98a0-f457940b468f-kube-api-access-5b4jq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:28.056205 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.056175    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fc4bb-d514-46e7-98a0-f457940b468f-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:28.302929 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.302838    5196 generic.go:356] "Generic (PLEG): container finished" podID="f681aac0-0c6c-4dec-b8ea-9b837683c320" containerID="cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10" exitCode=0
Nov 03 21:16:28.302929 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.302899    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"
Nov 03 21:16:28.303200 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.302925    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c" event={"ID":"f681aac0-0c6c-4dec-b8ea-9b837683c320","Type":"ContainerDied","Data":"cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10"}
Nov 03 21:16:28.303200 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.303020    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c" event={"ID":"f681aac0-0c6c-4dec-b8ea-9b837683c320","Type":"ContainerDied","Data":"663776cd6fcfc8faadd8c2997dfbfd3e8351d5373805ce3a24e54096a0b08ab5"}
Nov 03 21:16:28.303200 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.303054    5196 scope.go:122] "RemoveContainer" containerID="cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10"
Nov 03 21:16:28.305312 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.305237    5196 generic.go:356] "Generic (PLEG): container finished" podID="898fc4bb-d514-46e7-98a0-f457940b468f" containerID="70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b" exitCode=0
Nov 03 21:16:28.305581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.305325    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"
Nov 03 21:16:28.305581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.305346    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx" event={"ID":"898fc4bb-d514-46e7-98a0-f457940b468f","Type":"ContainerDied","Data":"70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b"}
Nov 03 21:16:28.305581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.305396    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx" event={"ID":"898fc4bb-d514-46e7-98a0-f457940b468f","Type":"ContainerDied","Data":"b8cfdef09f1daad00c17618f4d68dc16e6e6feb88d40825085a12f6179698b10"}
Nov 03 21:16:28.315185 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.315167    5196 scope.go:122] "RemoveContainer" containerID="cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10"
Nov 03 21:16:28.315533 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:16:28.315511    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10\": container with ID starting with cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10 not found: ID does not exist" containerID="cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10"
Nov 03 21:16:28.315617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.315541    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10"} err="failed to get container status \"cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10\": rpc error: code = NotFound desc = could not find container \"cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10\": container with ID starting with cddaa735eebe51c0ecf8407105a00a7392432a979ce9730bc606512d263f6a10 not found: ID does not exist"
Nov 03 21:16:28.315617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.315559    5196 scope.go:122] "RemoveContainer" containerID="70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b"
Nov 03 21:16:28.326526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.326505    5196 scope.go:122] "RemoveContainer" containerID="70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b"
Nov 03 21:16:28.326862 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:16:28.326842    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b\": container with ID starting with 70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b not found: ID does not exist" containerID="70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b"
Nov 03 21:16:28.326944 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.326871    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b"} err="failed to get container status \"70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b\": rpc error: code = NotFound desc = could not find container \"70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b\": container with ID starting with 70c29aa27098d45fef427b0ea0af854526766641e418cc3a33874cc71095a17b not found: ID does not exist"
Nov 03 21:16:28.435656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.435618    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"]
Nov 03 21:16:28.442086 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:16:28.442048    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7b4055_35a6_45a0_8de9_5367671311ab.slice/crio-8cb60f4354a2e82103b843ed85952f3ad6240067303d06eeaf184547f0fa9778 WatchSource:0}: Error finding container 8cb60f4354a2e82103b843ed85952f3ad6240067303d06eeaf184547f0fa9778: Status 404 returned error can't find the container with id 8cb60f4354a2e82103b843ed85952f3ad6240067303d06eeaf184547f0fa9778
Nov 03 21:16:28.461257 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.461224    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"]
Nov 03 21:16:28.466156 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.466123    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-562eb2-7c6c96cf95-k6zgx"]
Nov 03 21:16:28.524853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.524821    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:28.571942 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.571892    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"]
Nov 03 21:16:28.575878 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.575844    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-562eb2-67c47f6dcf-bg57c"]
Nov 03 21:16:28.672918 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.672872    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:28.861535 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:28.861503    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:29.309838 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.309690    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:29.314068 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.314032    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx" event={"ID":"aa7b4055-35a6-45a0-8de9-5367671311ab","Type":"ContainerStarted","Data":"ac498e729594284677d6da077a0615c972eca916418d145b88ca16c7c75f40e0"}
Nov 03 21:16:29.314258 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.314085    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx" event={"ID":"aa7b4055-35a6-45a0-8de9-5367671311ab","Type":"ContainerStarted","Data":"8cb60f4354a2e82103b843ed85952f3ad6240067303d06eeaf184547f0fa9778"}
Nov 03 21:16:29.425157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.425078    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx" podStartSLOduration=2.425061699 podStartE2EDuration="2.425061699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:29.423841048 +0000 UTC m=+2829.427799773" watchObservedRunningTime="2026-11-03 21:16:29.425061699 +0000 UTC m=+2829.429020423"
Nov 03 21:16:29.760631 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.760598    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"]
Nov 03 21:16:29.761210 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.761193    5196 cpu_manager.go:414] "containerMap: removing container" podUID="898fc4bb-d514-46e7-98a0-f457940b468f" containerName="payload-pre-processing"
Nov 03 21:16:29.761210 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.761208    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898fc4bb-d514-46e7-98a0-f457940b468f" containerName="payload-pre-processing"
Nov 03 21:16:29.761335 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.761224    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f681aac0-0c6c-4dec-b8ea-9b837683c320" containerName="payload-processing"
Nov 03 21:16:29.761335 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.761229    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f681aac0-0c6c-4dec-b8ea-9b837683c320" containerName="payload-processing"
Nov 03 21:16:29.761408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.761350    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="898fc4bb-d514-46e7-98a0-f457940b468f" containerName="payload-pre-processing"
Nov 03 21:16:29.761408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.761362    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f681aac0-0c6c-4dec-b8ea-9b837683c320" containerName="payload-processing"
Nov 03 21:16:29.764733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.764713    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:29.817491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.817458    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"]
Nov 03 21:16:29.878357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.878325    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:29.878585 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.878370    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:29.878585 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.878391    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44dl\" (UniqueName: \"kubernetes.io/projected/57e1d50b-e835-4086-8739-240f82ed902e-kube-api-access-l44dl\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:29.979931 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.979896    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:29.980201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.979954    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:29.980201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.979986    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44dl\" (UniqueName: \"kubernetes.io/projected/57e1d50b-e835-4086-8739-240f82ed902e-kube-api-access-l44dl\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:29.980771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.980743    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:29.982510 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:29.982491    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:30.038334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:30.038258    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44dl\" (UniqueName: \"kubernetes.io/projected/57e1d50b-e835-4086-8739-240f82ed902e-kube-api-access-l44dl\") pod \"maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:30.078739 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:30.078699    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:30.111150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:30.111088    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="898fc4bb-d514-46e7-98a0-f457940b468f" path="/var/lib/kubelet/pods/898fc4bb-d514-46e7-98a0-f457940b468f/volumes"
Nov 03 21:16:30.111583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:30.111568    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f681aac0-0c6c-4dec-b8ea-9b837683c320" path="/var/lib/kubelet/pods/f681aac0-0c6c-4dec-b8ea-9b837683c320/volumes"
Nov 03 21:16:30.324782 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:30.324749    5196 generic.go:356] "Generic (PLEG): container finished" podID="aa7b4055-35a6-45a0-8de9-5367671311ab" containerID="ac498e729594284677d6da077a0615c972eca916418d145b88ca16c7c75f40e0" exitCode=0
Nov 03 21:16:30.324991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:30.324844    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx" event={"ID":"aa7b4055-35a6-45a0-8de9-5367671311ab","Type":"ContainerDied","Data":"ac498e729594284677d6da077a0615c972eca916418d145b88ca16c7c75f40e0"}
Nov 03 21:16:30.460631 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:30.460599    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"]
Nov 03 21:16:30.467585 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:16:30.467538    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e1d50b_e835_4086_8739_240f82ed902e.slice/crio-8929ee416b825ccd9165ea48ce18c9093ee1a6be6a038555e84793ee0c668cf2 WatchSource:0}: Error finding container 8929ee416b825ccd9165ea48ce18c9093ee1a6be6a038555e84793ee0c668cf2: Status 404 returned error can't find the container with id 8929ee416b825ccd9165ea48ce18c9093ee1a6be6a038555e84793ee0c668cf2
Nov 03 21:16:31.334138 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.334083    5196 generic.go:356] "Generic (PLEG): container finished" podID="57e1d50b-e835-4086-8739-240f82ed902e" containerID="36b3587492276131cf66ad59c25f859f795129cdcccbbd060695b55ccece71e5" exitCode=0
Nov 03 21:16:31.334596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.334231    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn" event={"ID":"57e1d50b-e835-4086-8739-240f82ed902e","Type":"ContainerDied","Data":"36b3587492276131cf66ad59c25f859f795129cdcccbbd060695b55ccece71e5"}
Nov 03 21:16:31.334596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.334312    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn" event={"ID":"57e1d50b-e835-4086-8739-240f82ed902e","Type":"ContainerStarted","Data":"8929ee416b825ccd9165ea48ce18c9093ee1a6be6a038555e84793ee0c668cf2"}
Nov 03 21:16:31.483726 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.483692    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:31.600724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.599589    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-openshift-ca\") pod \"aa7b4055-35a6-45a0-8de9-5367671311ab\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") "
Nov 03 21:16:31.600724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.599697    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-xks-ca\") pod \"aa7b4055-35a6-45a0-8de9-5367671311ab\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") "
Nov 03 21:16:31.600724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.599788    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7b4055-35a6-45a0-8de9-5367671311ab-kube-api-access-pjkx6\" (UniqueName: \"kubernetes.io/projected/aa7b4055-35a6-45a0-8de9-5367671311ab-kube-api-access-pjkx6\") pod \"aa7b4055-35a6-45a0-8de9-5367671311ab\" (UID: \"aa7b4055-35a6-45a0-8de9-5367671311ab\") "
Nov 03 21:16:31.601773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.601480    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-openshift-ca" pod "aa7b4055-35a6-45a0-8de9-5367671311ab" (UID: "aa7b4055-35a6-45a0-8de9-5367671311ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:31.603272 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.603247    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7b4055-35a6-45a0-8de9-5367671311ab-kube-api-access-pjkx6" pod "aa7b4055-35a6-45a0-8de9-5367671311ab" (UID: "aa7b4055-35a6-45a0-8de9-5367671311ab"). InnerVolumeSpecName "kube-api-access-pjkx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:31.603989 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.603962    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-xks-ca" pod "aa7b4055-35a6-45a0-8de9-5367671311ab" (UID: "aa7b4055-35a6-45a0-8de9-5367671311ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:31.701566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.701412    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:31.701566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.701448    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7b4055-35a6-45a0-8de9-5367671311ab-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:31.701566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.701464    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkx6\" (UniqueName: \"kubernetes.io/projected/aa7b4055-35a6-45a0-8de9-5367671311ab-kube-api-access-pjkx6\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:31.702576 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:31.702545    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:32.343030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.342998    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"
Nov 03 21:16:32.343030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.343031    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx" event={"ID":"aa7b4055-35a6-45a0-8de9-5367671311ab","Type":"ContainerDied","Data":"8cb60f4354a2e82103b843ed85952f3ad6240067303d06eeaf184547f0fa9778"}
Nov 03 21:16:32.343634 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.343136    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cb60f4354a2e82103b843ed85952f3ad6240067303d06eeaf184547f0fa9778"
Nov 03 21:16:32.519964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.519932    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:32.622688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.612894    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-xks-ca\") pod \"57e1d50b-e835-4086-8739-240f82ed902e\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") "
Nov 03 21:16:32.622688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.612979    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-openshift-ca\") pod \"57e1d50b-e835-4086-8739-240f82ed902e\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") "
Nov 03 21:16:32.622688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.613023    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e1d50b-e835-4086-8739-240f82ed902e-kube-api-access-l44dl\" (UniqueName: \"kubernetes.io/projected/57e1d50b-e835-4086-8739-240f82ed902e-kube-api-access-l44dl\") pod \"57e1d50b-e835-4086-8739-240f82ed902e\" (UID: \"57e1d50b-e835-4086-8739-240f82ed902e\") "
Nov 03 21:16:32.622688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.616821    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e1d50b-e835-4086-8739-240f82ed902e-kube-api-access-l44dl" pod "57e1d50b-e835-4086-8739-240f82ed902e" (UID: "57e1d50b-e835-4086-8739-240f82ed902e"). InnerVolumeSpecName "kube-api-access-l44dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:32.622688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.619552    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-xks-ca" pod "57e1d50b-e835-4086-8739-240f82ed902e" (UID: "57e1d50b-e835-4086-8739-240f82ed902e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:32.622688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.619882    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-openshift-ca" pod "57e1d50b-e835-4086-8739-240f82ed902e" (UID: "57e1d50b-e835-4086-8739-240f82ed902e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:32.714872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.714779    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:32.714872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.714814    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e1d50b-e835-4086-8739-240f82ed902e-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:32.714872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:32.714825    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l44dl\" (UniqueName: \"kubernetes.io/projected/57e1d50b-e835-4086-8739-240f82ed902e-kube-api-access-l44dl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:33.352145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:33.352083    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"
Nov 03 21:16:33.352565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:33.352121    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn" event={"ID":"57e1d50b-e835-4086-8739-240f82ed902e","Type":"ContainerDied","Data":"8929ee416b825ccd9165ea48ce18c9093ee1a6be6a038555e84793ee0c668cf2"}
Nov 03 21:16:33.352565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:33.352247    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8929ee416b825ccd9165ea48ce18c9093ee1a6be6a038555e84793ee0c668cf2"
Nov 03 21:16:34.268651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:34.268608    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"]
Nov 03 21:16:34.374820 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:34.374770    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:36.895956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:36.895924    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:39.138286 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:39.138251    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"]
Nov 03 21:16:39.246119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:39.246072    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"]
Nov 03 21:16:39.519898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:39.519798    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:39.668304 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:39.668272    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:41.586386 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:41.586343    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"]
Nov 03 21:16:41.694964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:41.694933    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"]
Nov 03 21:16:41.905020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:41.904903    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:42.094285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:42.094248    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:42.243959 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:42.243856    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:44.139500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.139439    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6" podUID="d7efd8bf-e3ab-4041-91cf-5719ee35c4b6" containerName="payload-processing" containerID="cri-o://3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f" gracePeriod=25
Nov 03 21:16:44.247214 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.247169    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2" podUID="082ac835-40a4-4990-865f-91397c4bad21" containerName="payload-pre-processing" containerID="cri-o://4d370ed8f4c3f318597520fe3c6b87ade206aeaf2f3fbc5be44331381338bc59" gracePeriod=25
Nov 03 21:16:44.422244 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.422080    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 21:16:44.437164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.437125    5196 generic.go:356] "Generic (PLEG): container finished" podID="d7efd8bf-e3ab-4041-91cf-5719ee35c4b6" containerID="3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f" exitCode=0
Nov 03 21:16:44.437417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.437389    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6" event={"ID":"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6","Type":"ContainerDied","Data":"3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f"}
Nov 03 21:16:44.437494 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.437457    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6" event={"ID":"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6","Type":"ContainerDied","Data":"48d4e8e7dc5833e656f06279ff615ad610154f4a43e4ffa57bccd4f1d7e60423"}
Nov 03 21:16:44.437494 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.437488    5196 scope.go:122] "RemoveContainer" containerID="3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f"
Nov 03 21:16:44.437662 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.437648    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"
Nov 03 21:16:44.440808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.440771    5196 generic.go:356] "Generic (PLEG): container finished" podID="082ac835-40a4-4990-865f-91397c4bad21" containerID="4d370ed8f4c3f318597520fe3c6b87ade206aeaf2f3fbc5be44331381338bc59" exitCode=0
Nov 03 21:16:44.440932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.440821    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2" event={"ID":"082ac835-40a4-4990-865f-91397c4bad21","Type":"ContainerDied","Data":"4d370ed8f4c3f318597520fe3c6b87ade206aeaf2f3fbc5be44331381338bc59"}
Nov 03 21:16:44.452238 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.452213    5196 scope.go:122] "RemoveContainer" containerID="3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f"
Nov 03 21:16:44.452648 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:16:44.452615    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f\": container with ID starting with 3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f not found: ID does not exist" containerID="3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f"
Nov 03 21:16:44.452745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.452659    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f"} err="failed to get container status \"3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f\": rpc error: code = NotFound desc = could not find container \"3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f\": container with ID starting with 3a5aba51133b6b4b373ccecb012dba5e3db39bda0d2753f683736debc1c4165f not found: ID does not exist"
Nov 03 21:16:44.503351 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.503320    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 21:16:44.552417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.552371    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-kube-api-access-vg8q5\" (UniqueName: \"kubernetes.io/projected/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-kube-api-access-vg8q5\") pod \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\" (UID: \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\") "
Nov 03 21:16:44.552417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.552421    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-plugins-config-volume\") pod \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\" (UID: \"d7efd8bf-e3ab-4041-91cf-5719ee35c4b6\") "
Nov 03 21:16:44.552908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.552875    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-plugins-config-volume" pod "d7efd8bf-e3ab-4041-91cf-5719ee35c4b6" (UID: "d7efd8bf-e3ab-4041-91cf-5719ee35c4b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:44.555431 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.555395    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-kube-api-access-vg8q5" pod "d7efd8bf-e3ab-4041-91cf-5719ee35c4b6" (UID: "d7efd8bf-e3ab-4041-91cf-5719ee35c4b6"). InnerVolumeSpecName "kube-api-access-vg8q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:44.654049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.653952    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/082ac835-40a4-4990-865f-91397c4bad21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082ac835-40a4-4990-865f-91397c4bad21-plugins-config-volume\") pod \"082ac835-40a4-4990-865f-91397c4bad21\" (UID: \"082ac835-40a4-4990-865f-91397c4bad21\") "
Nov 03 21:16:44.654049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.653996    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082ac835-40a4-4990-865f-91397c4bad21-kube-api-access-8vxfb\" (UniqueName: \"kubernetes.io/projected/082ac835-40a4-4990-865f-91397c4bad21-kube-api-access-8vxfb\") pod \"082ac835-40a4-4990-865f-91397c4bad21\" (UID: \"082ac835-40a4-4990-865f-91397c4bad21\") "
Nov 03 21:16:44.654421 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.654406    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:44.654508 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.654423    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg8q5\" (UniqueName: \"kubernetes.io/projected/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6-kube-api-access-vg8q5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:44.654508 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.654473    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082ac835-40a4-4990-865f-91397c4bad21-plugins-config-volume" pod "082ac835-40a4-4990-865f-91397c4bad21" (UID: "082ac835-40a4-4990-865f-91397c4bad21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:44.656424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.656394    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082ac835-40a4-4990-865f-91397c4bad21-kube-api-access-8vxfb" pod "082ac835-40a4-4990-865f-91397c4bad21" (UID: "082ac835-40a4-4990-865f-91397c4bad21"). InnerVolumeSpecName "kube-api-access-8vxfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:44.755686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.755650    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082ac835-40a4-4990-865f-91397c4bad21-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:44.755686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.755678    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxfb\" (UniqueName: \"kubernetes.io/projected/082ac835-40a4-4990-865f-91397c4bad21-kube-api-access-8vxfb\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:44.854332 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.854293    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"]
Nov 03 21:16:44.857323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:44.857286    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-45f792-57b8644f69-zdbs6"]
Nov 03 21:16:45.449107 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:45.448986    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"
Nov 03 21:16:45.449107 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:45.449023    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2" event={"ID":"082ac835-40a4-4990-865f-91397c4bad21","Type":"ContainerDied","Data":"7d417864f7e1c1a0ba3a9e4bd4cdf445926cc86632d0dc0fb4e5fdb236561742"}
Nov 03 21:16:45.449662 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:45.449083    5196 scope.go:122] "RemoveContainer" containerID="4d370ed8f4c3f318597520fe3c6b87ade206aeaf2f3fbc5be44331381338bc59"
Nov 03 21:16:45.605677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:45.605641    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"]
Nov 03 21:16:45.652493 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:45.652443    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-45f792-774995dccf-njbk2"]
Nov 03 21:16:46.109203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.109161    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082ac835-40a4-4990-865f-91397c4bad21" path="/var/lib/kubelet/pods/082ac835-40a4-4990-865f-91397c4bad21/volumes"
Nov 03 21:16:46.109667 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.109645    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7efd8bf-e3ab-4041-91cf-5719ee35c4b6" path="/var/lib/kubelet/pods/d7efd8bf-e3ab-4041-91cf-5719ee35c4b6/volumes"
Nov 03 21:16:46.587705 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.587667    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l" podUID="3dd9390b-e208-479c-b6c7-a52c4ddbff51" containerName="payload-processing" containerID="cri-o://8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595" gracePeriod=25
Nov 03 21:16:46.657659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.657621    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:46.696332 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.696278    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt" podUID="46df0c79-9c14-40f7-9ece-6577e91d8dfc" containerName="payload-pre-processing" containerID="cri-o://99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4" gracePeriod=25
Nov 03 21:16:46.864942 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.864912    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 21:16:46.971380 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.971350    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 21:16:46.981824 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.981790    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd9390b-e208-479c-b6c7-a52c4ddbff51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd9390b-e208-479c-b6c7-a52c4ddbff51-plugins-config-volume\") pod \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\" (UID: \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\") "
Nov 03 21:16:46.982016 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.981836    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd9390b-e208-479c-b6c7-a52c4ddbff51-kube-api-access-w7lxc\" (UniqueName: \"kubernetes.io/projected/3dd9390b-e208-479c-b6c7-a52c4ddbff51-kube-api-access-w7lxc\") pod \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\" (UID: \"3dd9390b-e208-479c-b6c7-a52c4ddbff51\") "
Nov 03 21:16:46.982266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.982237    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd9390b-e208-479c-b6c7-a52c4ddbff51-plugins-config-volume" pod "3dd9390b-e208-479c-b6c7-a52c4ddbff51" (UID: "3dd9390b-e208-479c-b6c7-a52c4ddbff51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:46.984251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:46.984226    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd9390b-e208-479c-b6c7-a52c4ddbff51-kube-api-access-w7lxc" pod "3dd9390b-e208-479c-b6c7-a52c4ddbff51" (UID: "3dd9390b-e208-479c-b6c7-a52c4ddbff51"). InnerVolumeSpecName "kube-api-access-w7lxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:47.084683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.084650    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46df0c79-9c14-40f7-9ece-6577e91d8dfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46df0c79-9c14-40f7-9ece-6577e91d8dfc-plugins-config-volume\") pod \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\" (UID: \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\") "
Nov 03 21:16:47.084858 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.084703    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46df0c79-9c14-40f7-9ece-6577e91d8dfc-kube-api-access-lwztq\" (UniqueName: \"kubernetes.io/projected/46df0c79-9c14-40f7-9ece-6577e91d8dfc-kube-api-access-lwztq\") pod \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\" (UID: \"46df0c79-9c14-40f7-9ece-6577e91d8dfc\") "
Nov 03 21:16:47.084986 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.084971    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd9390b-e208-479c-b6c7-a52c4ddbff51-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:47.085054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.084989    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7lxc\" (UniqueName: \"kubernetes.io/projected/3dd9390b-e208-479c-b6c7-a52c4ddbff51-kube-api-access-w7lxc\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:47.085214 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.085187    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46df0c79-9c14-40f7-9ece-6577e91d8dfc-plugins-config-volume" pod "46df0c79-9c14-40f7-9ece-6577e91d8dfc" (UID: "46df0c79-9c14-40f7-9ece-6577e91d8dfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:47.087372 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.087344    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46df0c79-9c14-40f7-9ece-6577e91d8dfc-kube-api-access-lwztq" pod "46df0c79-9c14-40f7-9ece-6577e91d8dfc" (UID: "46df0c79-9c14-40f7-9ece-6577e91d8dfc"). InnerVolumeSpecName "kube-api-access-lwztq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:47.186651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.186562    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46df0c79-9c14-40f7-9ece-6577e91d8dfc-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:47.186651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.186594    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwztq\" (UniqueName: \"kubernetes.io/projected/46df0c79-9c14-40f7-9ece-6577e91d8dfc-kube-api-access-lwztq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:47.468686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.468591    5196 generic.go:356] "Generic (PLEG): container finished" podID="46df0c79-9c14-40f7-9ece-6577e91d8dfc" containerID="99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4" exitCode=0
Nov 03 21:16:47.468686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.468660    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"
Nov 03 21:16:47.468686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.468672    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt" event={"ID":"46df0c79-9c14-40f7-9ece-6577e91d8dfc","Type":"ContainerDied","Data":"99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4"}
Nov 03 21:16:47.468940 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.468765    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt" event={"ID":"46df0c79-9c14-40f7-9ece-6577e91d8dfc","Type":"ContainerDied","Data":"9045b29075c8df57c7beddad4b98700840df0e7d41aa4d341fc8e094961eca0e"}
Nov 03 21:16:47.468940 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.468794    5196 scope.go:122] "RemoveContainer" containerID="99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4"
Nov 03 21:16:47.470486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.470389    5196 generic.go:356] "Generic (PLEG): container finished" podID="3dd9390b-e208-479c-b6c7-a52c4ddbff51" containerID="8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595" exitCode=0
Nov 03 21:16:47.470486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.470452    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"
Nov 03 21:16:47.470486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.470485    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l" event={"ID":"3dd9390b-e208-479c-b6c7-a52c4ddbff51","Type":"ContainerDied","Data":"8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595"}
Nov 03 21:16:47.470486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.470539    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l" event={"ID":"3dd9390b-e208-479c-b6c7-a52c4ddbff51","Type":"ContainerDied","Data":"95cf9882123ef1ba2f035c3e503875724e9c875921c67f567c78cdf9051ee902"}
Nov 03 21:16:47.480151 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.480079    5196 scope.go:122] "RemoveContainer" containerID="99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4"
Nov 03 21:16:47.480469 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:16:47.480447    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4\": container with ID starting with 99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4 not found: ID does not exist" containerID="99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4"
Nov 03 21:16:47.480566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.480478    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4"} err="failed to get container status \"99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4\": rpc error: code = NotFound desc = could not find container \"99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4\": container with ID starting with 99535a6b52ef59ae48712850a5caef79e815f08868567de9c4a3721634aa62c4 not found: ID does not exist"
Nov 03 21:16:47.480566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.480503    5196 scope.go:122] "RemoveContainer" containerID="8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595"
Nov 03 21:16:47.490653 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.490631    5196 scope.go:122] "RemoveContainer" containerID="8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595"
Nov 03 21:16:47.490975 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:16:47.490952    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595\": container with ID starting with 8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595 not found: ID does not exist" containerID="8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595"
Nov 03 21:16:47.491027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.490989    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595"} err="failed to get container status \"8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595\": rpc error: code = NotFound desc = could not find container \"8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595\": container with ID starting with 8b6f1ee0c7b0cbdc379b8086a58edbef92f1ac31b0d40953bd8725226807f595 not found: ID does not exist"
Nov 03 21:16:47.624891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.624853    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"]
Nov 03 21:16:47.628052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.628024    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-984cbc-7bf58fc985-hb55l"]
Nov 03 21:16:47.729964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.729880    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"]
Nov 03 21:16:47.732936 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:47.732910    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-984cbc-667b6cfb75-hk7dt"]
Nov 03 21:16:48.109762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:48.109727    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd9390b-e208-479c-b6c7-a52c4ddbff51" path="/var/lib/kubelet/pods/3dd9390b-e208-479c-b6c7-a52c4ddbff51/volumes"
Nov 03 21:16:48.110224 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:48.110208    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46df0c79-9c14-40f7-9ece-6577e91d8dfc" path="/var/lib/kubelet/pods/46df0c79-9c14-40f7-9ece-6577e91d8dfc/volumes"
Nov 03 21:16:48.485932 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:16:48.485840    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:16:49.380531 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.380493    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" containerID="cri-o://5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4" gracePeriod=45
Nov 03 21:16:49.454672 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.454635    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": dial tcp 10.132.0.99:8000: connect: connection refused"
Nov 03 21:16:49.648657 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.648593    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 21:16:49.816952 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.816913    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-home\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-home\") pod \"6249b701-4a12-439f-b47d-287004b120b8\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") "
Nov 03 21:16:49.817201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.816980    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6249b701-4a12-439f-b47d-287004b120b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6249b701-4a12-439f-b47d-287004b120b8-tls-certs\") pod \"6249b701-4a12-439f-b47d-287004b120b8\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") "
Nov 03 21:16:49.817201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817067    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-dshm\") pod \"6249b701-4a12-439f-b47d-287004b120b8\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") "
Nov 03 21:16:49.817201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817133    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-tmp-dir\") pod \"6249b701-4a12-439f-b47d-287004b120b8\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") "
Nov 03 21:16:49.817401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817227    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6249b701-4a12-439f-b47d-287004b120b8-kube-api-access-t4jj6\" (UniqueName: \"kubernetes.io/projected/6249b701-4a12-439f-b47d-287004b120b8-kube-api-access-t4jj6\") pod \"6249b701-4a12-439f-b47d-287004b120b8\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") "
Nov 03 21:16:49.817401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817270    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-model-cache\") pod \"6249b701-4a12-439f-b47d-287004b120b8\" (UID: \"6249b701-4a12-439f-b47d-287004b120b8\") "
Nov 03 21:16:49.817401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817272    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-home" pod "6249b701-4a12-439f-b47d-287004b120b8" (UID: "6249b701-4a12-439f-b47d-287004b120b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:49.817575 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817550    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-tmp-dir" pod "6249b701-4a12-439f-b47d-287004b120b8" (UID: "6249b701-4a12-439f-b47d-287004b120b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:49.817796 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817768    5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-tmp-dir\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:49.817904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817805    5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-home\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:49.817904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.817773    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-model-cache" pod "6249b701-4a12-439f-b47d-287004b120b8" (UID: "6249b701-4a12-439f-b47d-287004b120b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:49.820438 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.820315    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6249b701-4a12-439f-b47d-287004b120b8-kube-api-access-t4jj6" pod "6249b701-4a12-439f-b47d-287004b120b8" (UID: "6249b701-4a12-439f-b47d-287004b120b8"). InnerVolumeSpecName "kube-api-access-t4jj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:49.820438 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.820404    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-dshm" pod "6249b701-4a12-439f-b47d-287004b120b8" (UID: "6249b701-4a12-439f-b47d-287004b120b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:49.820594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.820438    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6249b701-4a12-439f-b47d-287004b120b8-tls-certs" pod "6249b701-4a12-439f-b47d-287004b120b8" (UID: "6249b701-4a12-439f-b47d-287004b120b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:49.918567 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.918482    5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6249b701-4a12-439f-b47d-287004b120b8-tls-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:49.918567 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.918512    5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-dshm\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:49.918567 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.918523    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jj6\" (UniqueName: \"kubernetes.io/projected/6249b701-4a12-439f-b47d-287004b120b8-kube-api-access-t4jj6\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:49.918567 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:49.918534    5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6249b701-4a12-439f-b47d-287004b120b8-model-cache\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:50.498191 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.498158    5196 generic.go:356] "Generic (PLEG): container finished" podID="6249b701-4a12-439f-b47d-287004b120b8" containerID="5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4" exitCode=0
Nov 03 21:16:50.498703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.498209    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"
Nov 03 21:16:50.498703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.498250    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" event={"ID":"6249b701-4a12-439f-b47d-287004b120b8","Type":"ContainerDied","Data":"5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4"}
Nov 03 21:16:50.498703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.498311    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks" event={"ID":"6249b701-4a12-439f-b47d-287004b120b8","Type":"ContainerDied","Data":"e48b399b4b1087c6c29af4b1fed5454de2fc144e99fddb8e7bd49bd4cff4de76"}
Nov 03 21:16:50.498703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.498335    5196 scope.go:122] "RemoveContainer" containerID="5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4"
Nov 03 21:16:50.508652 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.508632    5196 scope.go:122] "RemoveContainer" containerID="5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4"
Nov 03 21:16:50.509000 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:16:50.508981    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4\": container with ID starting with 5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4 not found: ID does not exist" containerID="5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4"
Nov 03 21:16:50.509053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.509011    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4"} err="failed to get container status \"5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4\": rpc error: code = NotFound desc = could not find container \"5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4\": container with ID starting with 5e8394f9055aac7f3c4ec96adb365d2781e5be1cd474c668d29f880e273d7fd4 not found: ID does not exist"
Nov 03 21:16:50.655861 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.655825    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"]
Nov 03 21:16:50.702809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.702781    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-984cbc/premium-w4-984cbc-kserve-7c79f9db7f-9b7ks"]
Nov 03 21:16:50.830222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:50.830184    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:51.823840 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:51.823801    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:52.109036 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:52.108955    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6249b701-4a12-439f-b47d-287004b120b8" path="/var/lib/kubelet/pods/6249b701-4a12-439f-b47d-287004b120b8/volumes"
Nov 03 21:16:52.338918 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:52.338879    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:52.578191 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:52.578140    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:52.724324 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:52.724291    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:52.816500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:52.816465    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"]
Nov 03 21:16:52.863227 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:52.863146    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-45f792-bzvtx"]
Nov 03 21:16:53.230478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:53.230381    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:54.108565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:54.108526    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7b4055-35a6-45a0-8de9-5367671311ab" path="/var/lib/kubelet/pods/aa7b4055-35a6-45a0-8de9-5367671311ab/volumes"
Nov 03 21:16:56.045453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:56.045418    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"]
Nov 03 21:16:56.092297 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:56.092254    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a637bba-l6bpb"]
Nov 03 21:16:56.108852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:56.108822    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7" path="/var/lib/kubelet/pods/ba23cc96-8ad1-4ddc-bc1e-3578bb73afb7/volumes"
Nov 03 21:16:57.051211 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051177    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"]
Nov 03 21:16:57.051742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051721    5196 cpu_manager.go:414] "containerMap: removing container" podUID="3dd9390b-e208-479c-b6c7-a52c4ddbff51" containerName="payload-processing"
Nov 03 21:16:57.051742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051737    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd9390b-e208-479c-b6c7-a52c4ddbff51" containerName="payload-processing"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051752    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d7efd8bf-e3ab-4041-91cf-5719ee35c4b6" containerName="payload-processing"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051758    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7efd8bf-e3ab-4041-91cf-5719ee35c4b6" containerName="payload-processing"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051775    5196 cpu_manager.go:414] "containerMap: removing container" podUID="aa7b4055-35a6-45a0-8de9-5367671311ab" containerName="revoke-keys"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051779    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7b4055-35a6-45a0-8de9-5367671311ab" containerName="revoke-keys"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051793    5196 cpu_manager.go:414] "containerMap: removing container" podUID="57e1d50b-e835-4086-8739-240f82ed902e" containerName="revoke-keys"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051798    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e1d50b-e835-4086-8739-240f82ed902e" containerName="revoke-keys"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051809    5196 cpu_manager.go:414] "containerMap: removing container" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051813    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051821    5196 cpu_manager.go:414] "containerMap: removing container" podUID="082ac835-40a4-4990-865f-91397c4bad21" containerName="payload-pre-processing"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051825    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082ac835-40a4-4990-865f-91397c4bad21" containerName="payload-pre-processing"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051836    5196 cpu_manager.go:414] "containerMap: removing container" podUID="46df0c79-9c14-40f7-9ece-6577e91d8dfc" containerName="payload-pre-processing"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051840    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46df0c79-9c14-40f7-9ece-6577e91d8dfc" containerName="payload-pre-processing"
Nov 03 21:16:57.051911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051916    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd9390b-e208-479c-b6c7-a52c4ddbff51" containerName="payload-processing"
Nov 03 21:16:57.052402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051925    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7efd8bf-e3ab-4041-91cf-5719ee35c4b6" containerName="payload-processing"
Nov 03 21:16:57.052402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051933    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7b4055-35a6-45a0-8de9-5367671311ab" containerName="revoke-keys"
Nov 03 21:16:57.052402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051941    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e1d50b-e835-4086-8739-240f82ed902e" containerName="revoke-keys"
Nov 03 21:16:57.052402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051947    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6249b701-4a12-439f-b47d-287004b120b8" containerName="main"
Nov 03 21:16:57.052402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051953    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="082ac835-40a4-4990-865f-91397c4bad21" containerName="payload-pre-processing"
Nov 03 21:16:57.052402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.051960    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="46df0c79-9c14-40f7-9ece-6577e91d8dfc" containerName="payload-pre-processing"
Nov 03 21:16:57.057174 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.057155    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.097828 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.097789    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.098017 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.097944    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd54j\" (UniqueName: \"kubernetes.io/projected/a2287ee7-cce8-4670-907a-7b373f80382b-kube-api-access-pd54j\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.098068 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.098054    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.109979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.109947    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"]
Nov 03 21:16:57.152138 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.152087    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:16:57.152138 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.152131    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:16:57.155469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.155443    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"]
Nov 03 21:16:57.159803 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.159785    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.199038 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.198998    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.199248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.199046    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.199248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.199136    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.199248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.199204    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwfq\" (UniqueName: \"kubernetes.io/projected/f5497d40-7d72-46b9-9571-239251bb3ac9-kube-api-access-zmwfq\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.199248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.199224    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd54j\" (UniqueName: \"kubernetes.io/projected/a2287ee7-cce8-4670-907a-7b373f80382b-kube-api-access-pd54j\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.199248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.199244    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.199883 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.199856    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.201827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.201787    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.207867 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.207833    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"]
Nov 03 21:16:57.278013 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.277977    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd54j\" (UniqueName: \"kubernetes.io/projected/a2287ee7-cce8-4670-907a-7b373f80382b-kube-api-access-pd54j\") pod \"maas-api-revoke-sub-24402417-9bx9j\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.300455 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.300422    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.300697 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.300481    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwfq\" (UniqueName: \"kubernetes.io/projected/f5497d40-7d72-46b9-9571-239251bb3ac9-kube-api-access-zmwfq\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.300697 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.300502    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.301077 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.301055    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.303111 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.303054    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.354412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.354380    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwfq\" (UniqueName: \"kubernetes.io/projected/f5497d40-7d72-46b9-9571-239251bb3ac9-kube-api-access-zmwfq\") pod \"maas-api-revoke-sub-3a9432de-vwm7j\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.370490 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.370455    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:16:57.473704 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.473668    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:16:57.529453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.529411    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:16:57.767124 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.751397    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"]
Nov 03 21:16:57.860009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.859945    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"]
Nov 03 21:16:57.866508 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:16:57.866477    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5497d40_7d72_46b9_9571_239251bb3ac9.slice/crio-5502b300edb470e06ff540afe72fa7b3d711cf1e69d320592e53b4217274d4e0 WatchSource:0}: Error finding container 5502b300edb470e06ff540afe72fa7b3d711cf1e69d320592e53b4217274d4e0: Status 404 returned error can't find the container with id 5502b300edb470e06ff540afe72fa7b3d711cf1e69d320592e53b4217274d4e0
Nov 03 21:16:57.970513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:57.970462    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:58.566275 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:58.566236    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j" event={"ID":"f5497d40-7d72-46b9-9571-239251bb3ac9","Type":"ContainerStarted","Data":"03f590365a930404c6df2f48981b739817014c124382aa85bd39182c271cc8e0"}
Nov 03 21:16:58.566745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:58.566301    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j" event={"ID":"f5497d40-7d72-46b9-9571-239251bb3ac9","Type":"ContainerStarted","Data":"5502b300edb470e06ff540afe72fa7b3d711cf1e69d320592e53b4217274d4e0"}
Nov 03 21:16:58.568143 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:58.568115    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j" event={"ID":"a2287ee7-cce8-4670-907a-7b373f80382b","Type":"ContainerStarted","Data":"7451ab0be83a5d10c855d0b78ff61958d49b9a99db0f109a25216748adbd118e"}
Nov 03 21:16:58.568292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:58.568162    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j" event={"ID":"a2287ee7-cce8-4670-907a-7b373f80382b","Type":"ContainerStarted","Data":"12a0f8235c1dcc90cf487b7430550a82cd4f723121c92cf51380a662ac2bd576"}
Nov 03 21:16:58.677412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:58.677360    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j" podStartSLOduration=1.6773456150000001 podStartE2EDuration="1.677345615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:58.674754232 +0000 UTC m=+2858.678712968" watchObservedRunningTime="2026-11-03 21:16:58.677345615 +0000 UTC m=+2858.681304336"
Nov 03 21:16:58.782329 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:58.782276    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j" podStartSLOduration=1.782262737 podStartE2EDuration="1.782262737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:58.781610915 +0000 UTC m=+2858.785569662" watchObservedRunningTime="2026-11-03 21:16:58.782262737 +0000 UTC m=+2858.786221460"
Nov 03 21:16:59.200826 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:59.200789    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:59.415020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:59.414983    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:59.565885 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:59.565793    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:59.576498 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:59.576464    5196 generic.go:356] "Generic (PLEG): container finished" podID="f5497d40-7d72-46b9-9571-239251bb3ac9" containerID="03f590365a930404c6df2f48981b739817014c124382aa85bd39182c271cc8e0" exitCode=0
Nov 03 21:16:59.576981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:59.576553    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j" event={"ID":"f5497d40-7d72-46b9-9571-239251bb3ac9","Type":"ContainerDied","Data":"03f590365a930404c6df2f48981b739817014c124382aa85bd39182c271cc8e0"}
Nov 03 21:16:59.578467 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:59.578442    5196 generic.go:356] "Generic (PLEG): container finished" podID="a2287ee7-cce8-4670-907a-7b373f80382b" containerID="7451ab0be83a5d10c855d0b78ff61958d49b9a99db0f109a25216748adbd118e" exitCode=0
Nov 03 21:16:59.578601 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:16:59.578537    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j" event={"ID":"a2287ee7-cce8-4670-907a-7b373f80382b","Type":"ContainerDied","Data":"7451ab0be83a5d10c855d0b78ff61958d49b9a99db0f109a25216748adbd118e"}
Nov 03 21:17:00.765387 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.765348    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:17:00.793721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.793687    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:17:00.842877 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.842780    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5497d40-7d72-46b9-9571-239251bb3ac9-kube-api-access-zmwfq\" (UniqueName: \"kubernetes.io/projected/f5497d40-7d72-46b9-9571-239251bb3ac9-kube-api-access-zmwfq\") pod \"f5497d40-7d72-46b9-9571-239251bb3ac9\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") "
Nov 03 21:17:00.842877 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.842866    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-openshift-ca\") pod \"f5497d40-7d72-46b9-9571-239251bb3ac9\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") "
Nov 03 21:17:00.843192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.842931    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-xks-ca\") pod \"a2287ee7-cce8-4670-907a-7b373f80382b\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") "
Nov 03 21:17:00.843192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.843027    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2287ee7-cce8-4670-907a-7b373f80382b-kube-api-access-pd54j\" (UniqueName: \"kubernetes.io/projected/a2287ee7-cce8-4670-907a-7b373f80382b-kube-api-access-pd54j\") pod \"a2287ee7-cce8-4670-907a-7b373f80382b\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") "
Nov 03 21:17:00.843192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.843058    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-xks-ca\") pod \"f5497d40-7d72-46b9-9571-239251bb3ac9\" (UID: \"f5497d40-7d72-46b9-9571-239251bb3ac9\") "
Nov 03 21:17:00.843192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.843187    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-openshift-ca\") pod \"a2287ee7-cce8-4670-907a-7b373f80382b\" (UID: \"a2287ee7-cce8-4670-907a-7b373f80382b\") "
Nov 03 21:17:00.843428 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.843311    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-openshift-ca" pod "f5497d40-7d72-46b9-9571-239251bb3ac9" (UID: "f5497d40-7d72-46b9-9571-239251bb3ac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:00.843668 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.843640    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-openshift-ca" pod "a2287ee7-cce8-4670-907a-7b373f80382b" (UID: "a2287ee7-cce8-4670-907a-7b373f80382b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:00.843975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.843930    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:00.843975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.843955    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:00.845513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.845489    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-xks-ca" pod "a2287ee7-cce8-4670-907a-7b373f80382b" (UID: "a2287ee7-cce8-4670-907a-7b373f80382b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:00.845513 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.845498    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2287ee7-cce8-4670-907a-7b373f80382b-kube-api-access-pd54j" pod "a2287ee7-cce8-4670-907a-7b373f80382b" (UID: "a2287ee7-cce8-4670-907a-7b373f80382b"). InnerVolumeSpecName "kube-api-access-pd54j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:00.845910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.845882    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-xks-ca" pod "f5497d40-7d72-46b9-9571-239251bb3ac9" (UID: "f5497d40-7d72-46b9-9571-239251bb3ac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:00.846061 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.846040    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5497d40-7d72-46b9-9571-239251bb3ac9-kube-api-access-zmwfq" pod "f5497d40-7d72-46b9-9571-239251bb3ac9" (UID: "f5497d40-7d72-46b9-9571-239251bb3ac9"). InnerVolumeSpecName "kube-api-access-zmwfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:00.944809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.944766    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwfq\" (UniqueName: \"kubernetes.io/projected/f5497d40-7d72-46b9-9571-239251bb3ac9-kube-api-access-zmwfq\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:00.944809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.944802    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2287ee7-cce8-4670-907a-7b373f80382b-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:00.944809 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.944816    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd54j\" (UniqueName: \"kubernetes.io/projected/a2287ee7-cce8-4670-907a-7b373f80382b-kube-api-access-pd54j\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:00.945052 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:00.944829    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5497d40-7d72-46b9-9571-239251bb3ac9-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:01.594126 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:01.594053    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"
Nov 03 21:17:01.594126 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:01.594082    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j" event={"ID":"a2287ee7-cce8-4670-907a-7b373f80382b","Type":"ContainerDied","Data":"12a0f8235c1dcc90cf487b7430550a82cd4f723121c92cf51380a662ac2bd576"}
Nov 03 21:17:01.594361 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:01.594163    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12a0f8235c1dcc90cf487b7430550a82cd4f723121c92cf51380a662ac2bd576"
Nov 03 21:17:01.595903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:01.595872    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j" event={"ID":"f5497d40-7d72-46b9-9571-239251bb3ac9","Type":"ContainerDied","Data":"5502b300edb470e06ff540afe72fa7b3d711cf1e69d320592e53b4217274d4e0"}
Nov 03 21:17:01.596037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:01.595906    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"
Nov 03 21:17:01.596037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:01.595929    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5502b300edb470e06ff540afe72fa7b3d711cf1e69d320592e53b4217274d4e0"
Nov 03 21:17:02.916081 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:02.916046    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:07.981571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:07.981530    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:09.274705 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:09.274670    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:10.813948 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:10.813914    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:11.007204 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:11.007160    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:11.769843 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:11.769793    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:12.111896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:12.111858    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:12.283497 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:12.283458    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:12.434037 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:12.433935    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:13.637742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:13.637700    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:13.877041 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:13.877010    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:14.035520 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:14.035443    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:14.121349 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:14.121311    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"]
Nov 03 21:17:14.168055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:14.168024    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-984cbc-cs6jn"]
Nov 03 21:17:14.515528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:14.515488    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:16.109318 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:16.109274    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e1d50b-e835-4086-8739-240f82ed902e" path="/var/lib/kubelet/pods/57e1d50b-e835-4086-8739-240f82ed902e/volumes"
Nov 03 21:17:17.591850 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:17.591815    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:17:17.771880 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:17.771848    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:18.004566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:18.004461    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:18.485715 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:17:18.485673    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:17:18.972711 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:18.972675    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:20.909311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:20.909266    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:21.130453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:21.130409    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:21.172888 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:21.172800    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"]
Nov 03 21:17:21.173184 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:21.173147    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy" containerID="cri-o://a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33" gracePeriod=30
Nov 03 21:17:21.289278 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:21.289241    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:23.138488 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:23.138455    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:24.636137 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:24.636025    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:24.850191 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:24.850149    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:25.006875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:25.006779    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:25.611174 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:25.611086    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:26.436776 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.436749    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 21:17:26.521178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521028    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-podinfo\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521143    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-socket\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521232    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-token\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521257    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-credential-socket\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521288    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istiod-ca-cert\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521332    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-certs\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521364    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-kube-api-access-vrxk7\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-kube-api-access-vrxk7\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521454    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-socket" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.521734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521479    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-data\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521512    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-envoy\") pod \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\" (UID: \"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1\") "
Nov 03 21:17:26.521734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521558    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-credential-socket" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.521734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521659    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istiod-ca-cert" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:26.522007 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.521803    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-data" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.522070 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.522026    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.522148 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.522052    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.522207 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.522160    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-certs" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.522246 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.522176    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.522281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.522261    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.523836 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.523809    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-kube-api-access-vrxk7" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "kube-api-access-vrxk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:26.523836 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.523810    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-envoy" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.524133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.524107    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-podinfo" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:17:26.524230 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.524130    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-token" pod "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" (UID: "ac9f05b7-5f08-4c4f-ba48-4465dd06eef1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:26.622997 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.622961    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.622997 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.622989    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.622997 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.622999    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxk7\" (UniqueName: \"kubernetes.io/projected/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-kube-api-access-vrxk7\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.623310 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.623011    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.623310 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.623021    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.783584 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.783488    5196 generic.go:356] "Generic (PLEG): container finished" podID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerID="a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33" exitCode=0
Nov 03 21:17:26.783584 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.783548    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"
Nov 03 21:17:26.783584 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.783568    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" event={"ID":"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1","Type":"ContainerDied","Data":"a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33"}
Nov 03 21:17:26.783892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.783612    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k" event={"ID":"ac9f05b7-5f08-4c4f-ba48-4465dd06eef1","Type":"ContainerDied","Data":"8647d5a823c3b2f8259df1c0f48db278a7ed539b177a61a9daa635e786e26d1a"}
Nov 03 21:17:26.783892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.783635    5196 scope.go:122] "RemoveContainer" containerID="a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33"
Nov 03 21:17:26.795956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.795934    5196 scope.go:122] "RemoveContainer" containerID="a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33"
Nov 03 21:17:26.796298 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:17:26.796279    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33\": container with ID starting with a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33 not found: ID does not exist" containerID="a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33"
Nov 03 21:17:26.796394 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.796307    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33"} err="failed to get container status \"a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33\": rpc error: code = NotFound desc = could not find container \"a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33\": container with ID starting with a0d26f4aa71e04b2781f3b533b00d1915564e416154081de4f1ce60482479c33 not found: ID does not exist"
Nov 03 21:17:26.940986 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.940947    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"]
Nov 03 21:17:26.944583 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:26.944559    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-984cbc-openshift-default-77f5856fbc-mt55k"]
Nov 03 21:17:27.917126 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:27.917082    5196 scope.go:122] "RemoveContainer" containerID="9bea1cb8574ce3f71c22f4b1e7e4e5706698bf1281cba14adb1b65a292002903"
Nov 03 21:17:27.927684 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:27.927662    5196 scope.go:122] "RemoveContainer" containerID="9dc1ab8e59531f0f6b8365fbcf51d9657749552f98bdc40c1a13e7d75301e509"
Nov 03 21:17:28.108903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:28.108869    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" path="/var/lib/kubelet/pods/ac9f05b7-5f08-4c4f-ba48-4465dd06eef1/volumes"
Nov 03 21:17:29.872325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:29.872281    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:30.691416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:30.691387    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:17:31.316488 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:31.316442    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:31.520977 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:31.520939    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:31.674133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:31.674063    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:32.103764 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.103727    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:32.627218 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627178    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"]
Nov 03 21:17:32.627821 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627796    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a2287ee7-cce8-4670-907a-7b373f80382b" containerName="revoke-keys"
Nov 03 21:17:32.627821 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627817    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2287ee7-cce8-4670-907a-7b373f80382b" containerName="revoke-keys"
Nov 03 21:17:32.627994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627845    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f5497d40-7d72-46b9-9571-239251bb3ac9" containerName="revoke-keys"
Nov 03 21:17:32.627994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627850    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5497d40-7d72-46b9-9571-239251bb3ac9" containerName="revoke-keys"
Nov 03 21:17:32.627994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627870    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy"
Nov 03 21:17:32.627994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627875    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy"
Nov 03 21:17:32.627994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627961    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9f05b7-5f08-4c4f-ba48-4465dd06eef1" containerName="istio-proxy"
Nov 03 21:17:32.627994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627972    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2287ee7-cce8-4670-907a-7b373f80382b" containerName="revoke-keys"
Nov 03 21:17:32.627994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.627981    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5497d40-7d72-46b9-9571-239251bb3ac9" containerName="revoke-keys"
Nov 03 21:17:32.631189 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.631171    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.683451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.683410    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.683644 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.683549    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"]
Nov 03 21:17:32.683644 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.683633    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.683731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.683683    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/433910ae-1585-4ee9-9cc8-f34a306cce7c-kube-api-access-qpwks\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.726494 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.726451    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:17:32.726676 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.726507    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:17:32.784926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.784881    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/433910ae-1585-4ee9-9cc8-f34a306cce7c-kube-api-access-qpwks\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.785178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.784947    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.785178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.785056    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.785803 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.785779    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.787871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.787844    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.839299 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.839164    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/433910ae-1585-4ee9-9cc8-f34a306cce7c-kube-api-access-qpwks\") pod \"maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:32.945055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:32.944955    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:33.345040 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:33.345002    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"]
Nov 03 21:17:33.351716 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:17:33.351682    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433910ae_1585_4ee9_9cc8_f34a306cce7c.slice/crio-bd9511724fde1a035a1b288b469ece52437326d408a4db24a8d659376b17b19a WatchSource:0}: Error finding container bd9511724fde1a035a1b288b469ece52437326d408a4db24a8d659376b17b19a: Status 404 returned error can't find the container with id bd9511724fde1a035a1b288b469ece52437326d408a4db24a8d659376b17b19a
Nov 03 21:17:33.838245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:33.838210    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2" event={"ID":"433910ae-1585-4ee9-9cc8-f34a306cce7c","Type":"ContainerStarted","Data":"7c1e0f39bfcc4449c496c02e520c24816837b91024de9100bb34a7592287b791"}
Nov 03 21:17:33.838710 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:33.838266    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2" event={"ID":"433910ae-1585-4ee9-9cc8-f34a306cce7c","Type":"ContainerStarted","Data":"bd9511724fde1a035a1b288b469ece52437326d408a4db24a8d659376b17b19a"}
Nov 03 21:17:33.947594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:33.947477    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2" podStartSLOduration=1.947462101 podStartE2EDuration="1.947462101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:33.944777215 +0000 UTC m=+2893.948735939" watchObservedRunningTime="2026-11-03 21:17:33.947462101 +0000 UTC m=+2893.951420845"
Nov 03 21:17:34.857354 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:34.857311    5196 generic.go:356] "Generic (PLEG): container finished" podID="433910ae-1585-4ee9-9cc8-f34a306cce7c" containerID="7c1e0f39bfcc4449c496c02e520c24816837b91024de9100bb34a7592287b791" exitCode=0
Nov 03 21:17:34.857822 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:34.857448    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2" event={"ID":"433910ae-1585-4ee9-9cc8-f34a306cce7c","Type":"ContainerDied","Data":"7c1e0f39bfcc4449c496c02e520c24816837b91024de9100bb34a7592287b791"}
Nov 03 21:17:35.048415 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:35.048377    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:35.542006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:35.541971    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:35.780885 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:35.780848    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:36.018121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.018079    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:36.123981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.123898    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433910ae-1585-4ee9-9cc8-f34a306cce7c-kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/433910ae-1585-4ee9-9cc8-f34a306cce7c-kube-api-access-qpwks\") pod \"433910ae-1585-4ee9-9cc8-f34a306cce7c\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") "
Nov 03 21:17:36.124180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.124028    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-openshift-ca\") pod \"433910ae-1585-4ee9-9cc8-f34a306cce7c\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") "
Nov 03 21:17:36.124180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.124066    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-xks-ca\") pod \"433910ae-1585-4ee9-9cc8-f34a306cce7c\" (UID: \"433910ae-1585-4ee9-9cc8-f34a306cce7c\") "
Nov 03 21:17:36.124419 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.124385    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-openshift-ca" pod "433910ae-1585-4ee9-9cc8-f34a306cce7c" (UID: "433910ae-1585-4ee9-9cc8-f34a306cce7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:36.124555 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.124534    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:36.126612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.126581    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-xks-ca" pod "433910ae-1585-4ee9-9cc8-f34a306cce7c" (UID: "433910ae-1585-4ee9-9cc8-f34a306cce7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:36.128004 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.127976    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433910ae-1585-4ee9-9cc8-f34a306cce7c-kube-api-access-qpwks" pod "433910ae-1585-4ee9-9cc8-f34a306cce7c" (UID: "433910ae-1585-4ee9-9cc8-f34a306cce7c"). InnerVolumeSpecName "kube-api-access-qpwks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:36.225968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.225930    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433910ae-1585-4ee9-9cc8-f34a306cce7c-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:36.225968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.225960    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/433910ae-1585-4ee9-9cc8-f34a306cce7c-kube-api-access-qpwks\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:36.872565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.872534    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"
Nov 03 21:17:36.872762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.872538    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2" event={"ID":"433910ae-1585-4ee9-9cc8-f34a306cce7c","Type":"ContainerDied","Data":"bd9511724fde1a035a1b288b469ece52437326d408a4db24a8d659376b17b19a"}
Nov 03 21:17:36.872762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:36.872656    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9511724fde1a035a1b288b469ece52437326d408a4db24a8d659376b17b19a"
Nov 03 21:17:37.159650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:37.159548    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:39.959086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:39.959050    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:44.115282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.115248    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"]
Nov 03 21:17:44.115778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.115756    5196 cpu_manager.go:414] "containerMap: removing container" podUID="433910ae-1585-4ee9-9cc8-f34a306cce7c" containerName="revoke-keys"
Nov 03 21:17:44.115778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.115771    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433910ae-1585-4ee9-9cc8-f34a306cce7c" containerName="revoke-keys"
Nov 03 21:17:44.115889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.115880    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="433910ae-1585-4ee9-9cc8-f34a306cce7c" containerName="revoke-keys"
Nov 03 21:17:44.119154 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.119132    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.141755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.141726    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"]
Nov 03 21:17:44.172003 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.171969    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"]
Nov 03 21:17:44.185668 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.185634    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:17:44.185884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.185728    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:17:44.213329 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.213286    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.213329 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.213337    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7zg\" (UniqueName: \"kubernetes.io/projected/78afa92b-29ba-4a52-be75-8c7125ca7c7d-kube-api-access-zd7zg\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.213635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.213471    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.250284 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.250250    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"]
Nov 03 21:17:44.314378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.314343    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.314378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.314379    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7zg\" (UniqueName: \"kubernetes.io/projected/78afa92b-29ba-4a52-be75-8c7125ca7c7d-kube-api-access-zd7zg\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.314636 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.314435    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.315024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.315000    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.317134 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.317064    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.368566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.368482    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7zg\" (UniqueName: \"kubernetes.io/projected/78afa92b-29ba-4a52-be75-8c7125ca7c7d-kube-api-access-zd7zg\") pod \"maas-api-revoke-sub-6996b236-qmgmh\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.433231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.433194    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:44.526947 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.526912    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:44.673376 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.673284    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:44.828608 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.828574    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"]
Nov 03 21:17:44.835769 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:17:44.835738    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78afa92b_29ba_4a52_be75_8c7125ca7c7d.slice/crio-588cf0c65b5f84bb5470ab48c0a13983486ea1ff49daf49c60134bf86187477c WatchSource:0}: Error finding container 588cf0c65b5f84bb5470ab48c0a13983486ea1ff49daf49c60134bf86187477c: Status 404 returned error can't find the container with id 588cf0c65b5f84bb5470ab48c0a13983486ea1ff49daf49c60134bf86187477c
Nov 03 21:17:44.929632 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:44.929549    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh" event={"ID":"78afa92b-29ba-4a52-be75-8c7125ca7c7d","Type":"ContainerStarted","Data":"588cf0c65b5f84bb5470ab48c0a13983486ea1ff49daf49c60134bf86187477c"}
Nov 03 21:17:45.943171 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:45.943130    5196 generic.go:356] "Generic (PLEG): container finished" podID="78afa92b-29ba-4a52-be75-8c7125ca7c7d" containerID="050f42fb449502eba887a4e065bf8e60740540e401f2347b38c5ba4b6de5f7f5" exitCode=0
Nov 03 21:17:45.943662 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:45.943216    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh" event={"ID":"78afa92b-29ba-4a52-be75-8c7125ca7c7d","Type":"ContainerDied","Data":"050f42fb449502eba887a4e065bf8e60740540e401f2347b38c5ba4b6de5f7f5"}
Nov 03 21:17:47.097936 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.097909    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:47.243893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.243798    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78afa92b-29ba-4a52-be75-8c7125ca7c7d-kube-api-access-zd7zg\" (UniqueName: \"kubernetes.io/projected/78afa92b-29ba-4a52-be75-8c7125ca7c7d-kube-api-access-zd7zg\") pod \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") "
Nov 03 21:17:47.243893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.243837    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-xks-ca\") pod \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") "
Nov 03 21:17:47.243893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.243881    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-openshift-ca\") pod \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\" (UID: \"78afa92b-29ba-4a52-be75-8c7125ca7c7d\") "
Nov 03 21:17:47.244344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.244320    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-openshift-ca" pod "78afa92b-29ba-4a52-be75-8c7125ca7c7d" (UID: "78afa92b-29ba-4a52-be75-8c7125ca7c7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:47.246201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.246177    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78afa92b-29ba-4a52-be75-8c7125ca7c7d-kube-api-access-zd7zg" pod "78afa92b-29ba-4a52-be75-8c7125ca7c7d" (UID: "78afa92b-29ba-4a52-be75-8c7125ca7c7d"). InnerVolumeSpecName "kube-api-access-zd7zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:47.246201 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.246183    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-xks-ca" pod "78afa92b-29ba-4a52-be75-8c7125ca7c7d" (UID: "78afa92b-29ba-4a52-be75-8c7125ca7c7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:47.345280 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.345243    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7zg\" (UniqueName: \"kubernetes.io/projected/78afa92b-29ba-4a52-be75-8c7125ca7c7d-kube-api-access-zd7zg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:47.345280 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.345276    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:47.345280 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.345287    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78afa92b-29ba-4a52-be75-8c7125ca7c7d-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:47.960307 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.960281    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"
Nov 03 21:17:47.960511 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.960274    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh" event={"ID":"78afa92b-29ba-4a52-be75-8c7125ca7c7d","Type":"ContainerDied","Data":"588cf0c65b5f84bb5470ab48c0a13983486ea1ff49daf49c60134bf86187477c"}
Nov 03 21:17:47.960511 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:47.960422    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="588cf0c65b5f84bb5470ab48c0a13983486ea1ff49daf49c60134bf86187477c"
Nov 03 21:17:48.487850 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:17:48.487809    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:17:49.142301 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.142251    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8" podUID="ecef5666-3f31-4093-b950-4a21b3f6a147" containerName="payload-processing" containerID="cri-o://c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83" gracePeriod=25
Nov 03 21:17:49.250766 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.250704    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8" podUID="6a6e35fd-2d40-46b4-8655-b20c54443217" containerName="payload-pre-processing" containerID="cri-o://d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08" gracePeriod=25
Nov 03 21:17:49.427472 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.427439    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 21:17:49.530641 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.530613    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 21:17:49.583030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.582990    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecef5666-3f31-4093-b950-4a21b3f6a147-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecef5666-3f31-4093-b950-4a21b3f6a147-plugins-config-volume\") pod \"ecef5666-3f31-4093-b950-4a21b3f6a147\" (UID: \"ecef5666-3f31-4093-b950-4a21b3f6a147\") "
Nov 03 21:17:49.583276 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.583062    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecef5666-3f31-4093-b950-4a21b3f6a147-kube-api-access-mkgcs\" (UniqueName: \"kubernetes.io/projected/ecef5666-3f31-4093-b950-4a21b3f6a147-kube-api-access-mkgcs\") pod \"ecef5666-3f31-4093-b950-4a21b3f6a147\" (UID: \"ecef5666-3f31-4093-b950-4a21b3f6a147\") "
Nov 03 21:17:49.583504 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.583476    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecef5666-3f31-4093-b950-4a21b3f6a147-plugins-config-volume" pod "ecef5666-3f31-4093-b950-4a21b3f6a147" (UID: "ecef5666-3f31-4093-b950-4a21b3f6a147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:49.585486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.585454    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecef5666-3f31-4093-b950-4a21b3f6a147-kube-api-access-mkgcs" pod "ecef5666-3f31-4093-b950-4a21b3f6a147" (UID: "ecef5666-3f31-4093-b950-4a21b3f6a147"). InnerVolumeSpecName "kube-api-access-mkgcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:49.684417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.684313    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6e35fd-2d40-46b4-8655-b20c54443217-kube-api-access-sxk7s\" (UniqueName: \"kubernetes.io/projected/6a6e35fd-2d40-46b4-8655-b20c54443217-kube-api-access-sxk7s\") pod \"6a6e35fd-2d40-46b4-8655-b20c54443217\" (UID: \"6a6e35fd-2d40-46b4-8655-b20c54443217\") "
Nov 03 21:17:49.684417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.684396    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6e35fd-2d40-46b4-8655-b20c54443217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6e35fd-2d40-46b4-8655-b20c54443217-plugins-config-volume\") pod \"6a6e35fd-2d40-46b4-8655-b20c54443217\" (UID: \"6a6e35fd-2d40-46b4-8655-b20c54443217\") "
Nov 03 21:17:49.684852 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.684827    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecef5666-3f31-4093-b950-4a21b3f6a147-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:49.684923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.684859    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkgcs\" (UniqueName: \"kubernetes.io/projected/ecef5666-3f31-4093-b950-4a21b3f6a147-kube-api-access-mkgcs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:49.684923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.684831    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6e35fd-2d40-46b4-8655-b20c54443217-plugins-config-volume" pod "6a6e35fd-2d40-46b4-8655-b20c54443217" (UID: "6a6e35fd-2d40-46b4-8655-b20c54443217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:49.686757 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.686728    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6e35fd-2d40-46b4-8655-b20c54443217-kube-api-access-sxk7s" pod "6a6e35fd-2d40-46b4-8655-b20c54443217" (UID: "6a6e35fd-2d40-46b4-8655-b20c54443217"). InnerVolumeSpecName "kube-api-access-sxk7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:49.785535 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.785498    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxk7s\" (UniqueName: \"kubernetes.io/projected/6a6e35fd-2d40-46b4-8655-b20c54443217-kube-api-access-sxk7s\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:49.785535 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.785532    5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6e35fd-2d40-46b4-8655-b20c54443217-plugins-config-volume\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:49.978164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.978053    5196 generic.go:356] "Generic (PLEG): container finished" podID="ecef5666-3f31-4093-b950-4a21b3f6a147" containerID="c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83" exitCode=0
Nov 03 21:17:49.978357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.978156    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8" event={"ID":"ecef5666-3f31-4093-b950-4a21b3f6a147","Type":"ContainerDied","Data":"c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83"}
Nov 03 21:17:49.978357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.978177    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"
Nov 03 21:17:49.978357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.978250    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8" event={"ID":"ecef5666-3f31-4093-b950-4a21b3f6a147","Type":"ContainerDied","Data":"5157e1caa4596273c47bce87884185b772e4d115dcdb460a030147e80b2fa48f"}
Nov 03 21:17:49.978357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.978286    5196 scope.go:122] "RemoveContainer" containerID="c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83"
Nov 03 21:17:49.980266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.980239    5196 generic.go:356] "Generic (PLEG): container finished" podID="6a6e35fd-2d40-46b4-8655-b20c54443217" containerID="d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08" exitCode=0
Nov 03 21:17:49.980408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.980287    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8" event={"ID":"6a6e35fd-2d40-46b4-8655-b20c54443217","Type":"ContainerDied","Data":"d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08"}
Nov 03 21:17:49.980408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.980318    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8" event={"ID":"6a6e35fd-2d40-46b4-8655-b20c54443217","Type":"ContainerDied","Data":"4387db8707206bc27f176aef69d388c13f12490d47d76b3236a586688e3b3627"}
Nov 03 21:17:49.980408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.980318    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"
Nov 03 21:17:49.990237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.990211    5196 scope.go:122] "RemoveContainer" containerID="c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83"
Nov 03 21:17:49.990640 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:17:49.990609    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83\": container with ID starting with c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83 not found: ID does not exist" containerID="c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83"
Nov 03 21:17:49.990759 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.990653    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83"} err="failed to get container status \"c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83\": rpc error: code = NotFound desc = could not find container \"c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83\": container with ID starting with c96a6398b12e330ac9854ae88c136538261e853f17c9f88b68ec04644f2b5f83 not found: ID does not exist"
Nov 03 21:17:49.990759 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:49.990678    5196 scope.go:122] "RemoveContainer" containerID="d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08"
Nov 03 21:17:50.002892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:50.002870    5196 scope.go:122] "RemoveContainer" containerID="d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08"
Nov 03 21:17:50.003303 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:17:50.003278    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08\": container with ID starting with d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08 not found: ID does not exist" containerID="d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08"
Nov 03 21:17:50.003393 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:50.003316    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08"} err="failed to get container status \"d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08\": rpc error: code = NotFound desc = could not find container \"d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08\": container with ID starting with d682f06793e597af70fa56feaa1bff428f1719ba0e442d8dde43ebcdec7cdc08 not found: ID does not exist"
Nov 03 21:17:50.133633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:50.133593    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"]
Nov 03 21:17:50.180429 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:50.180389    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fe2825-54b87d559-wctb8"]
Nov 03 21:17:50.246075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:50.245992    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"]
Nov 03 21:17:50.251200 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:50.251165    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fe2825-5f8cf8d7df-dtdq8"]
Nov 03 21:17:52.108768 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:52.108731    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6e35fd-2d40-46b4-8655-b20c54443217" path="/var/lib/kubelet/pods/6a6e35fd-2d40-46b4-8655-b20c54443217/volumes"
Nov 03 21:17:52.109198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:52.109147    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecef5666-3f31-4093-b950-4a21b3f6a147" path="/var/lib/kubelet/pods/ecef5666-3f31-4093-b950-4a21b3f6a147/volumes"
Nov 03 21:17:52.165287 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:52.165252    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:52.361947 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:52.361857    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:55.043573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:55.043538    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"]
Nov 03 21:17:55.054659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:55.054625    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c7089a-4tgcl"]
Nov 03 21:17:55.479347 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:55.479312    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:55.647786 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:55.647746    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:56.050727 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:56.050691    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"]
Nov 03 21:17:56.097802 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:56.097762    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596e08cc-pgrcd"]
Nov 03 21:17:56.108258 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:56.108226    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63935a9f-8280-4535-b770-c77191bf297b" path="/var/lib/kubelet/pods/63935a9f-8280-4535-b770-c77191bf297b/volumes"
Nov 03 21:17:56.108745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:56.108728    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7bf1caa-27f5-4366-89b5-cc88e17b58fa" path="/var/lib/kubelet/pods/e7bf1caa-27f5-4366-89b5-cc88e17b58fa/volumes"
Nov 03 21:17:58.882907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:58.882871    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:59.975507 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:17:59.975467    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:18:00.421808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:00.421771    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:00.696632 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:00.696530    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:00.894963 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:00.894927    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"]
Nov 03 21:18:00.941400 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:00.941362    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fe2825-97nw2"]
Nov 03 21:18:01.344359 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:01.344323    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:02.108740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:02.108706    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433910ae-1585-4ee9-9cc8-f34a306cce7c" path="/var/lib/kubelet/pods/433910ae-1585-4ee9-9cc8-f34a306cce7c/volumes"
Nov 03 21:18:02.753854 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:02.753824    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"]
Nov 03 21:18:02.754190 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:02.754145    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" containerID="cri-o://bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b" gracePeriod=30
Nov 03 21:18:06.990501 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:06.990461    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:07.189575 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:07.189509    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:07.347497 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:07.347463    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:07.432586 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:07.432551    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qnkkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:18:07.432767 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:07.432622    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:18:08.029836 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.029764    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:18:08.082923 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.082882    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-podinfo\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.082977    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-token\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-token\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083034    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-kube-api-access-wqccg\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-kube-api-access-wqccg\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083067    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-credential-socket\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083144    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-certs\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083176    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-data\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083222    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-socket\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083328    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-envoy\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083364    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istiod-ca-cert\") pod \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\" (UID: \"9a05f2b3-49f5-4ee3-b07b-555e4da13d13\") "
Nov 03 21:18:08.083525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083508    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-data" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:08.083720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083498    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-certs" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:08.083720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083567    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-credential-socket" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:08.083720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083640    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-socket" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:08.083908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083884    5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-credential-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.083964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083916    5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.083964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083931    5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-data\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.083964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083944    5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-workload-socket\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.084166 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.083991    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istiod-ca-cert" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:18:08.085665 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.085641    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-podinfo" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:18:08.085808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.085663    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-token" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:08.085808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.085725    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-envoy" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:08.085892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.085872    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-kube-api-access-wqccg" pod "9a05f2b3-49f5-4ee3-b07b-555e4da13d13" (UID: "9a05f2b3-49f5-4ee3-b07b-555e4da13d13"). InnerVolumeSpecName "kube-api-access-wqccg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:08.118657 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.118629    5196 generic.go:356] "Generic (PLEG): container finished" podID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerID="bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b" exitCode=0
Nov 03 21:18:08.118790 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.118669    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"
Nov 03 21:18:08.118790 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.118692    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" event={"ID":"9a05f2b3-49f5-4ee3-b07b-555e4da13d13","Type":"ContainerDied","Data":"bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b"}
Nov 03 21:18:08.118790 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.118737    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw" event={"ID":"9a05f2b3-49f5-4ee3-b07b-555e4da13d13","Type":"ContainerDied","Data":"135ce48e7745e8285102f9f267aeddccb64e76b34ee3ec5e89df96c60fbdcfcd"}
Nov 03 21:18:08.118790 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.118760    5196 scope.go:122] "RemoveContainer" containerID="bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b"
Nov 03 21:18:08.129526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.129507    5196 scope.go:122] "RemoveContainer" containerID="bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b"
Nov 03 21:18:08.129841 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:18:08.129823    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b\": container with ID starting with bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b not found: ID does not exist" containerID="bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b"
Nov 03 21:18:08.129884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.129851    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b"} err="failed to get container status \"bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b\": rpc error: code = NotFound desc = could not find container \"bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b\": container with ID starting with bf86d82db8f26a02c3e27e58185f179cc2186a85564064d0a8bf47234d6c7e0b not found: ID does not exist"
Nov 03 21:18:08.185666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.185580    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqccg\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-kube-api-access-wqccg\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.185666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.185619    5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-envoy\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.185666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.185637    5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istiod-ca-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.185666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.185667    5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-podinfo\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.186076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.185690    5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a05f2b3-49f5-4ee3-b07b-555e4da13d13-istio-token\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:08.294430 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.294337    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"]
Nov 03 21:18:08.297316 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:08.297285    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qnkkw"]
Nov 03 21:18:10.110663 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:10.110625    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" path="/var/lib/kubelet/pods/9a05f2b3-49f5-4ee3-b07b-555e4da13d13/volumes"
Nov 03 21:18:10.754395 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:10.754339    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:18:12.445407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:12.445366    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8f44875fd-nh9tp"]
Nov 03 21:18:12.445888 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:12.445587    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8f44875fd-nh9tp" podUID="9626020c-90dc-43d9-b77f-d512892db1ce" containerName="manager" containerID="cri-o://75557eb4230eb616b92797f6a2469851e48a84204705a0534fb4712145d4cf01" gracePeriod=30
Nov 03 21:18:13.157665 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.157634    5196 generic.go:356] "Generic (PLEG): container finished" podID="9626020c-90dc-43d9-b77f-d512892db1ce" containerID="75557eb4230eb616b92797f6a2469851e48a84204705a0534fb4712145d4cf01" exitCode=0
Nov 03 21:18:13.157851 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.157700    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-nh9tp" event={"ID":"9626020c-90dc-43d9-b77f-d512892db1ce","Type":"ContainerDied","Data":"75557eb4230eb616b92797f6a2469851e48a84204705a0534fb4712145d4cf01"}
Nov 03 21:18:13.210588 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.210555    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 21:18:13.341886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.341683    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9626020c-90dc-43d9-b77f-d512892db1ce-kube-api-access-whd4v\" (UniqueName: \"kubernetes.io/projected/9626020c-90dc-43d9-b77f-d512892db1ce-kube-api-access-whd4v\") pod \"9626020c-90dc-43d9-b77f-d512892db1ce\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") "
Nov 03 21:18:13.341886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.341778    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-metrics-certs\") pod \"9626020c-90dc-43d9-b77f-d512892db1ce\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") "
Nov 03 21:18:13.341886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.341850    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-webhook-cert\") pod \"9626020c-90dc-43d9-b77f-d512892db1ce\" (UID: \"9626020c-90dc-43d9-b77f-d512892db1ce\") "
Nov 03 21:18:13.344346 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.344319    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-metrics-certs" pod "9626020c-90dc-43d9-b77f-d512892db1ce" (UID: "9626020c-90dc-43d9-b77f-d512892db1ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:18:13.344668 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.344633    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9626020c-90dc-43d9-b77f-d512892db1ce-kube-api-access-whd4v" pod "9626020c-90dc-43d9-b77f-d512892db1ce" (UID: "9626020c-90dc-43d9-b77f-d512892db1ce"). InnerVolumeSpecName "kube-api-access-whd4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:13.344793 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.344751    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-webhook-cert" pod "9626020c-90dc-43d9-b77f-d512892db1ce" (UID: "9626020c-90dc-43d9-b77f-d512892db1ce"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:18:13.443579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.443546    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whd4v\" (UniqueName: \"kubernetes.io/projected/9626020c-90dc-43d9-b77f-d512892db1ce-kube-api-access-whd4v\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:13.443579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.443576    5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-metrics-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:13.443579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.443588    5196 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9626020c-90dc-43d9-b77f-d512892db1ce-webhook-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:13.915381 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:13.915341    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:14.118249 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:14.118218    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:14.166018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:14.165945    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-nh9tp"
Nov 03 21:18:14.166018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:14.165982    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-nh9tp" event={"ID":"9626020c-90dc-43d9-b77f-d512892db1ce","Type":"ContainerDied","Data":"3be6948e671df043b9ea68bb2e2576dc0b5753ca22283a4961dfffa51184a471"}
Nov 03 21:18:14.166263 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:14.166048    5196 scope.go:122] "RemoveContainer" containerID="75557eb4230eb616b92797f6a2469851e48a84204705a0534fb4712145d4cf01"
Nov 03 21:18:14.321780 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:14.321741    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8f44875fd-nh9tp"]
Nov 03 21:18:14.368721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:14.368688    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8f44875fd-nh9tp"]
Nov 03 21:18:14.416151 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:14.416048    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:15.142913 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.142872    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8f44875fd-sfzfk"]
Nov 03 21:18:15.143628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143605    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9626020c-90dc-43d9-b77f-d512892db1ce" containerName="manager"
Nov 03 21:18:15.143732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143630    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9626020c-90dc-43d9-b77f-d512892db1ce" containerName="manager"
Nov 03 21:18:15.143732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143648    5196 cpu_manager.go:414] "containerMap: removing container" podUID="78afa92b-29ba-4a52-be75-8c7125ca7c7d" containerName="revoke-keys"
Nov 03 21:18:15.143732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143656    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78afa92b-29ba-4a52-be75-8c7125ca7c7d" containerName="revoke-keys"
Nov 03 21:18:15.143732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143671    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ecef5666-3f31-4093-b950-4a21b3f6a147" containerName="payload-processing"
Nov 03 21:18:15.143732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143680    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecef5666-3f31-4093-b950-4a21b3f6a147" containerName="payload-processing"
Nov 03 21:18:15.143732 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143726    5196 cpu_manager.go:414] "containerMap: removing container" podUID="6a6e35fd-2d40-46b4-8655-b20c54443217" containerName="payload-pre-processing"
Nov 03 21:18:15.144048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143734    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6e35fd-2d40-46b4-8655-b20c54443217" containerName="payload-pre-processing"
Nov 03 21:18:15.144048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143747    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy"
Nov 03 21:18:15.144048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143754    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy"
Nov 03 21:18:15.144048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143900    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9626020c-90dc-43d9-b77f-d512892db1ce" containerName="manager"
Nov 03 21:18:15.144048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143912    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="78afa92b-29ba-4a52-be75-8c7125ca7c7d" containerName="revoke-keys"
Nov 03 21:18:15.144048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143923    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecef5666-3f31-4093-b950-4a21b3f6a147" containerName="payload-processing"
Nov 03 21:18:15.144048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143937    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6e35fd-2d40-46b4-8655-b20c54443217" containerName="payload-pre-processing"
Nov 03 21:18:15.144048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.143945    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a05f2b3-49f5-4ee3-b07b-555e4da13d13" containerName="istio-proxy"
Nov 03 21:18:15.148902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.148881    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.198574 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.198539    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 21:18:15.245770 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.245710    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 21:18:15.245770 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.245727    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 21:18:15.245770 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.245715    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrkqw\""
Nov 03 21:18:15.248796 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.248768    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8f44875fd-sfzfk"]
Nov 03 21:18:15.263485 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.263451    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-webhook-cert\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.263623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.263588    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqvw\" (UniqueName: \"kubernetes.io/projected/57372c12-bc46-4e59-ab70-9f621330b4dc-kube-api-access-lzqvw\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.263667 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.263640    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-metrics-certs\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.365361 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.365314    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-webhook-cert\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.365565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.365405    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqvw\" (UniqueName: \"kubernetes.io/projected/57372c12-bc46-4e59-ab70-9f621330b4dc-kube-api-access-lzqvw\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.365565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.365443    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-metrics-certs\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.368178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.368147    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-webhook-cert\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.368178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.368175    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-metrics-certs\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.419805 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.419734    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqvw\" (UniqueName: \"kubernetes.io/projected/57372c12-bc46-4e59-ab70-9f621330b4dc-kube-api-access-lzqvw\") pod \"maas-controller-8f44875fd-sfzfk\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") " pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.462470 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.462429    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:15.847531 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.847496    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8f44875fd-sfzfk"]
Nov 03 21:18:15.855776 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:18:15.855743    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57372c12_bc46_4e59_ab70_9f621330b4dc.slice/crio-400b90163c1765957f0440afba9a5aac47be5349200708551ced9a5eb0a37f18 WatchSource:0}: Error finding container 400b90163c1765957f0440afba9a5aac47be5349200708551ced9a5eb0a37f18: Status 404 returned error can't find the container with id 400b90163c1765957f0440afba9a5aac47be5349200708551ced9a5eb0a37f18
Nov 03 21:18:15.857281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:15.857261    5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:18:16.116948 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:16.116863    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9626020c-90dc-43d9-b77f-d512892db1ce" path="/var/lib/kubelet/pods/9626020c-90dc-43d9-b77f-d512892db1ce/volumes"
Nov 03 21:18:16.184794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:16.184752    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-sfzfk" event={"ID":"57372c12-bc46-4e59-ab70-9f621330b4dc","Type":"ContainerStarted","Data":"400b90163c1765957f0440afba9a5aac47be5349200708551ced9a5eb0a37f18"}
Nov 03 21:18:17.194579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:17.194477    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-sfzfk" event={"ID":"57372c12-bc46-4e59-ab70-9f621330b4dc","Type":"ContainerStarted","Data":"8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea"}
Nov 03 21:18:17.194997 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:17.194639    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:17.306459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:17.306394    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8f44875fd-sfzfk" podStartSLOduration=1.330122228 podStartE2EDuration="2.306375211s" totalImagesPullingTime="976.252983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:17.30297318 +0000 UTC m=+2937.306931937" watchObservedRunningTime="2026-11-03 21:18:17.306375211 +0000 UTC m=+2937.310333938"
Nov 03 21:18:18.502376 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:18:18.502330    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:18:18.951314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:18.951264    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rrffz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:18:18.951549 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:18.951336    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" podUID="e67311d6-9cf7-4985-abb1-1cc9763e6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:18:19.843881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:19.843842    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:28.007221 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:28.007175    5196 scope.go:122] "RemoveContainer" containerID="26b6ce8239ba350f72376285dffbadb6e36feb9af9317c76ba68d205bd68385f"
Nov 03 21:18:28.027295 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:28.027261    5196 scope.go:122] "RemoveContainer" containerID="db0c930d98a5db4f263c2ae6da2b14d34dab2534c978314531961d1a19a79eb8"
Nov 03 21:18:28.208731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:28.208701    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:28.428710 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:28.428674    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:48.486212 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:18:48.486170    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:18:49.299733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.299697    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:49.414348 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.414316    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8f44875fd-sfzfk"]
Nov 03 21:18:49.414578 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.414550    5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8f44875fd-sfzfk" podUID="57372c12-bc46-4e59-ab70-9f621330b4dc" containerName="manager" containerID="cri-o://8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea" gracePeriod=30
Nov 03 21:18:49.675990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.675964    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:49.754880 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.754842    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-metrics-certs\") pod \"57372c12-bc46-4e59-ab70-9f621330b4dc\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") "
Nov 03 21:18:49.755058 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.754945    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57372c12-bc46-4e59-ab70-9f621330b4dc-kube-api-access-lzqvw\" (UniqueName: \"kubernetes.io/projected/57372c12-bc46-4e59-ab70-9f621330b4dc-kube-api-access-lzqvw\") pod \"57372c12-bc46-4e59-ab70-9f621330b4dc\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") "
Nov 03 21:18:49.755058 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.754988    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-webhook-cert\") pod \"57372c12-bc46-4e59-ab70-9f621330b4dc\" (UID: \"57372c12-bc46-4e59-ab70-9f621330b4dc\") "
Nov 03 21:18:49.757247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.757208    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-webhook-cert" pod "57372c12-bc46-4e59-ab70-9f621330b4dc" (UID: "57372c12-bc46-4e59-ab70-9f621330b4dc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:18:49.757247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.757213    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57372c12-bc46-4e59-ab70-9f621330b4dc-kube-api-access-lzqvw" pod "57372c12-bc46-4e59-ab70-9f621330b4dc" (UID: "57372c12-bc46-4e59-ab70-9f621330b4dc"). InnerVolumeSpecName "kube-api-access-lzqvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:49.757458 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.757303    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-metrics-certs" pod "57372c12-bc46-4e59-ab70-9f621330b4dc" (UID: "57372c12-bc46-4e59-ab70-9f621330b4dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:18:49.856215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.856133    5196 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-webhook-cert\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:49.856215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.856165    5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57372c12-bc46-4e59-ab70-9f621330b4dc-metrics-certs\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:49.856215 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:49.856181    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzqvw\" (UniqueName: \"kubernetes.io/projected/57372c12-bc46-4e59-ab70-9f621330b4dc-kube-api-access-lzqvw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:50.424161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.424054    5196 generic.go:356] "Generic (PLEG): container finished" podID="57372c12-bc46-4e59-ab70-9f621330b4dc" containerID="8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea" exitCode=0
Nov 03 21:18:50.424161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.424117    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-sfzfk" event={"ID":"57372c12-bc46-4e59-ab70-9f621330b4dc","Type":"ContainerDied","Data":"8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea"}
Nov 03 21:18:50.424161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.424134    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-sfzfk"
Nov 03 21:18:50.424161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.424159    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-sfzfk" event={"ID":"57372c12-bc46-4e59-ab70-9f621330b4dc","Type":"ContainerDied","Data":"400b90163c1765957f0440afba9a5aac47be5349200708551ced9a5eb0a37f18"}
Nov 03 21:18:50.424485 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.424181    5196 scope.go:122] "RemoveContainer" containerID="8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea"
Nov 03 21:18:50.434407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.434385    5196 scope.go:122] "RemoveContainer" containerID="8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea"
Nov 03 21:18:50.434730 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:18:50.434681    5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea\": container with ID starting with 8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea not found: ID does not exist" containerID="8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea"
Nov 03 21:18:50.434829 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.434735    5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea"} err="failed to get container status \"8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea\": rpc error: code = NotFound desc = could not find container \"8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea\": container with ID starting with 8cf79164fcfebdc97852b7b33ef3168a54a89d6062a79e10ce4247375c081dea not found: ID does not exist"
Nov 03 21:18:50.579906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.579870    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8f44875fd-sfzfk"]
Nov 03 21:18:50.626754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:50.626717    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8f44875fd-sfzfk"]
Nov 03 21:18:51.844339 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:51.844299    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:51.957127 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:51.957080    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8f44875fd-kwb2n"]
Nov 03 21:18:51.957636 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:51.957623    5196 cpu_manager.go:414] "containerMap: removing container" podUID="57372c12-bc46-4e59-ab70-9f621330b4dc" containerName="manager"
Nov 03 21:18:51.957687 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:51.957638    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57372c12-bc46-4e59-ab70-9f621330b4dc" containerName="manager"
Nov 03 21:18:51.957736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:51.957727    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="57372c12-bc46-4e59-ab70-9f621330b4dc" containerName="manager"
Nov 03 21:18:51.962306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:51.962281    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.058762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.058726    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 21:18:52.058762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.058726    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 21:18:52.059019 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.058726    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrkqw\""
Nov 03 21:18:52.059019 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.058737    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 21:18:52.063725 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.063697    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8f44875fd-kwb2n"]
Nov 03 21:18:52.080022 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.079983    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ee1ede-2dda-4079-80e3-e14ecf2790b5-metrics-certs\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.080212 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.080142    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ee1ede-2dda-4079-80e3-e14ecf2790b5-webhook-cert\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.080305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.080289    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnrn\" (UniqueName: \"kubernetes.io/projected/67ee1ede-2dda-4079-80e3-e14ecf2790b5-kube-api-access-xcnrn\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.110660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.110585    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57372c12-bc46-4e59-ab70-9f621330b4dc" path="/var/lib/kubelet/pods/57372c12-bc46-4e59-ab70-9f621330b4dc/volumes"
Nov 03 21:18:52.181374 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.181328    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ee1ede-2dda-4079-80e3-e14ecf2790b5-metrics-certs\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.181600 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.181520    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ee1ede-2dda-4079-80e3-e14ecf2790b5-webhook-cert\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.181679 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.181625    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnrn\" (UniqueName: \"kubernetes.io/projected/67ee1ede-2dda-4079-80e3-e14ecf2790b5-kube-api-access-xcnrn\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.184076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.184046    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ee1ede-2dda-4079-80e3-e14ecf2790b5-metrics-certs\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.184239 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.184149    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ee1ede-2dda-4079-80e3-e14ecf2790b5-webhook-cert\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.235216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.235177    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnrn\" (UniqueName: \"kubernetes.io/projected/67ee1ede-2dda-4079-80e3-e14ecf2790b5-kube-api-access-xcnrn\") pod \"maas-controller-8f44875fd-kwb2n\" (UID: \"67ee1ede-2dda-4079-80e3-e14ecf2790b5\") " pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.275742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.275711    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:52.511376 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.511284    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:52.653266 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:52.653224    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8f44875fd-kwb2n"]
Nov 03 21:18:52.659733 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:18:52.659697    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ee1ede_2dda_4079_80e3_e14ecf2790b5.slice/crio-cfd72be65184be587a7b1a066bf5d47b8b0cb6cdf52dad06c6eca5b67a4dc43f WatchSource:0}: Error finding container cfd72be65184be587a7b1a066bf5d47b8b0cb6cdf52dad06c6eca5b67a4dc43f: Status 404 returned error can't find the container with id cfd72be65184be587a7b1a066bf5d47b8b0cb6cdf52dad06c6eca5b67a4dc43f
Nov 03 21:18:53.446983 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:53.446951    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-kwb2n" event={"ID":"67ee1ede-2dda-4079-80e3-e14ecf2790b5","Type":"ContainerStarted","Data":"cfd72be65184be587a7b1a066bf5d47b8b0cb6cdf52dad06c6eca5b67a4dc43f"}
Nov 03 21:18:54.455483 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:54.455384    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f44875fd-kwb2n" event={"ID":"67ee1ede-2dda-4079-80e3-e14ecf2790b5","Type":"ContainerStarted","Data":"d2071cde06fdddd0d0ad21ca449d49d0bcfe86cf1d341041c95b5df03416d7da"}
Nov 03 21:18:54.455483 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:54.455456    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:18:54.564787 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:18:54.564724    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8f44875fd-kwb2n" podStartSLOduration=2.62847641 podStartE2EDuration="3.564710606s" totalImagesPullingTime="936.234196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:54.562794199 +0000 UTC m=+2974.566752922" watchObservedRunningTime="2026-11-03 21:18:54.564710606 +0000 UTC m=+2974.568669329"
Nov 03 21:19:03.217439 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.217399    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"]
Nov 03 21:19:03.221936 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.221909    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.296995    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.297034    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.297060    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.297286    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.297315    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.297337    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.297377    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx7bm\" (UniqueName: \"kubernetes.io/projected/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-kube-api-access-sx7bm\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297424 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.297416    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.297571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.297499    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.317170 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.317137    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"]
Nov 03 21:19:03.398592 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398551    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.398592 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398593    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.398845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398613    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.398845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398676    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.398845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398696    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.398845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398726    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.398845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398751    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx7bm\" (UniqueName: \"kubernetes.io/projected/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-kube-api-access-sx7bm\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.398845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398794    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.399233 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.398848    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.399233 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.399219    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.399355 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.399248    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.399355 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.399319    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.399355 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.399343    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.399577 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.399549    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.401378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.401351    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.401589 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.401572    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.452568 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.452531    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.452733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.452587    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx7bm\" (UniqueName: \"kubernetes.io/projected/3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc-kube-api-access-sx7bm\") pod \"maas-default-gateway-openshift-default-696f47c6c-tcqsm\" (UID: \"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.535926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.535841    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:03.968784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:03.968754    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"]
Nov 03 21:19:04.017590 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:19:04.017557    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1e7367_a2c0_4190_9b38_20d1dcfb0ddc.slice/crio-d8dd4614dbccf0201614052aa3e82ff37d05b175ac34499826f70497582bfadd WatchSource:0}: Error finding container d8dd4614dbccf0201614052aa3e82ff37d05b175ac34499826f70497582bfadd: Status 404 returned error can't find the container with id d8dd4614dbccf0201614052aa3e82ff37d05b175ac34499826f70497582bfadd
Nov 03 21:19:04.019769 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:04.019737    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:19:04.019873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:04.019807    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:19:04.019873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:04.019853    5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:19:04.528152 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:04.528110    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" event={"ID":"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc","Type":"ContainerStarted","Data":"a7f31bfed3f25c1a7b908c3285fb7d34b31f94965b41939800c3d8b5b47a2700"}
Nov 03 21:19:04.528576 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:04.528186    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" event={"ID":"3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc","Type":"ContainerStarted","Data":"d8dd4614dbccf0201614052aa3e82ff37d05b175ac34499826f70497582bfadd"}
Nov 03 21:19:04.541503 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:04.541460    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:04.734169 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:04.734068    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" podStartSLOduration=1.734053731 podStartE2EDuration="1.734053731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:04.684295434 +0000 UTC m=+2984.688254217" watchObservedRunningTime="2026-11-03 21:19:04.734053731 +0000 UTC m=+2984.738012530"
Nov 03 21:19:05.466893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:05.466857    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8f44875fd-kwb2n"
Nov 03 21:19:05.560904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:05.560866    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-tcqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Nov 03 21:19:05.561479 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:05.560925    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" podUID="3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Nov 03 21:19:06.559785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:06.559262    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-tcqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Nov 03 21:19:06.559785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:06.559326    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" podUID="3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Nov 03 21:19:07.559293 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:07.559249    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-tcqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Nov 03 21:19:07.559701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:07.559312    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" podUID="3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Nov 03 21:19:08.551631 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:08.551588    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-tcqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Nov 03 21:19:08.551956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:08.551926    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" podUID="3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Nov 03 21:19:09.559124 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:09.558193    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-tcqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Nov 03 21:19:09.559124 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:09.558259    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" podUID="3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Nov 03 21:19:10.537241 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:10.537197    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-tcqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Nov 03 21:19:10.537478 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:10.537260    5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" podUID="3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Nov 03 21:19:12.273877 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:12.273839    5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:12.274922 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:12.274887    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:12.275613 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:12.275593    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm"
Nov 03 21:19:18.486212 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:19:18.486172    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:19:20.694798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:20.694766    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:19:20.701196 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:20.701158    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:19:20.713178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:20.713149    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:19:20.718813 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:20.718792    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:19:27.043910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:27.043869    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"]
Nov 03 21:19:27.090311 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:27.090276    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00886018-xnh9l"]
Nov 03 21:19:28.043454 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:28.043410    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"]
Nov 03 21:19:28.082338 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:28.082306    5196 scope.go:122] "RemoveContainer" containerID="81e69aeae209f5f16b65a7411303daf854b980c65a09d2441d29cc879d6b5254"
Nov 03 21:19:28.090838 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:28.090801    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52da04e1-zmjtz"]
Nov 03 21:19:28.093235 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:28.093215    5196 scope.go:122] "RemoveContainer" containerID="eee804c0c91444c6b166db4240cd1596878b1139b0c67c707017feaf2772c609"
Nov 03 21:19:28.116570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:28.116544    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501a5ad9-269c-4fac-be10-bfa97a1f52c8" path="/var/lib/kubelet/pods/501a5ad9-269c-4fac-be10-bfa97a1f52c8/volumes"
Nov 03 21:19:28.117019 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:28.117005    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebce0159-24c9-4a04-9125-6122296d96b3" path="/var/lib/kubelet/pods/ebce0159-24c9-4a04-9125-6122296d96b3/volumes"
Nov 03 21:19:37.550659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:37.550622    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:19:39.071121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:39.070420    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"]
Nov 03 21:19:39.110165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:39.108379    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e998d6-99jqg"]
Nov 03 21:19:40.117108 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:40.117063    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7fea920-3c6c-45dc-8f96-cb4c57ff175e" path="/var/lib/kubelet/pods/e7fea920-3c6c-45dc-8f96-cb4c57ff175e/volumes"
Nov 03 21:19:45.709198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.709162    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"]
Nov 03 21:19:45.713682 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.713662    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.763323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.763284    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"]
Nov 03 21:19:45.788911 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.788870    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgs2w\" (UniqueName: \"kubernetes.io/projected/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-kube-api-access-wgs2w\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.789087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.788925    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.789087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.789025    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.806906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.806866    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:19:45.807085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.806940    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:19:45.889934 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.889890    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgs2w\" (UniqueName: \"kubernetes.io/projected/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-kube-api-access-wgs2w\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.890159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.889952    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.890159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.889984    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.890765 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.890743    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.892792 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.892770    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:45.944416 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:45.944373    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgs2w\" (UniqueName: \"kubernetes.io/projected/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-kube-api-access-wgs2w\") pod \"maas-api-revoke-sub-c1e271a9-kmtjq\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:46.027527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:46.027444    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:46.404296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:46.404262    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"]
Nov 03 21:19:46.412959 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:19:46.412922    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472ad38a_7bd8_44e3_87ce_9ef3a84ee342.slice/crio-9ba455f41923ca15a5618299481ba615a0bd057abbd0542e7e50b99c35fe20e7 WatchSource:0}: Error finding container 9ba455f41923ca15a5618299481ba615a0bd057abbd0542e7e50b99c35fe20e7: Status 404 returned error can't find the container with id 9ba455f41923ca15a5618299481ba615a0bd057abbd0542e7e50b99c35fe20e7
Nov 03 21:19:46.885287 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:46.885247    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq" event={"ID":"472ad38a-7bd8-44e3-87ce-9ef3a84ee342","Type":"ContainerStarted","Data":"5d0ff92b70e1880d4d2991b50a7dd220962d08f29e86342ba4bec01cbce32597"}
Nov 03 21:19:46.885712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:46.885308    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq" event={"ID":"472ad38a-7bd8-44e3-87ce-9ef3a84ee342","Type":"ContainerStarted","Data":"9ba455f41923ca15a5618299481ba615a0bd057abbd0542e7e50b99c35fe20e7"}
Nov 03 21:19:46.994465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:46.994360    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq" podStartSLOduration=1.9943461120000001 podStartE2EDuration="1.994346112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:46.991817143 +0000 UTC m=+3026.995775886" watchObservedRunningTime="2026-11-03 21:19:46.994346112 +0000 UTC m=+3026.998304835"
Nov 03 21:19:47.894282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:47.894246    5196 generic.go:356] "Generic (PLEG): container finished" podID="472ad38a-7bd8-44e3-87ce-9ef3a84ee342" containerID="5d0ff92b70e1880d4d2991b50a7dd220962d08f29e86342ba4bec01cbce32597" exitCode=0
Nov 03 21:19:47.894717 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:47.894327    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq" event={"ID":"472ad38a-7bd8-44e3-87ce-9ef3a84ee342","Type":"ContainerDied","Data":"5d0ff92b70e1880d4d2991b50a7dd220962d08f29e86342ba4bec01cbce32597"}
Nov 03 21:19:48.486463 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:19:48.486413    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:19:49.053831 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.053806    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:49.123176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.123137    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-kube-api-access-wgs2w\" (UniqueName: \"kubernetes.io/projected/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-kube-api-access-wgs2w\") pod \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") "
Nov 03 21:19:49.123387 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.123231    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-openshift-ca\") pod \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") "
Nov 03 21:19:49.123387 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.123304    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-xks-ca\") pod \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\" (UID: \"472ad38a-7bd8-44e3-87ce-9ef3a84ee342\") "
Nov 03 21:19:49.123571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.123544    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-openshift-ca" pod "472ad38a-7bd8-44e3-87ce-9ef3a84ee342" (UID: "472ad38a-7bd8-44e3-87ce-9ef3a84ee342"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:19:49.123707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.123690    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:49.125516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.125491    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-kube-api-access-wgs2w" pod "472ad38a-7bd8-44e3-87ce-9ef3a84ee342" (UID: "472ad38a-7bd8-44e3-87ce-9ef3a84ee342"). InnerVolumeSpecName "kube-api-access-wgs2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:19:49.125635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.125536    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-xks-ca" pod "472ad38a-7bd8-44e3-87ce-9ef3a84ee342" (UID: "472ad38a-7bd8-44e3-87ce-9ef3a84ee342"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:19:49.224939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.224840    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgs2w\" (UniqueName: \"kubernetes.io/projected/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-kube-api-access-wgs2w\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:49.224939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.224876    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472ad38a-7bd8-44e3-87ce-9ef3a84ee342-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:49.909575 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.909537    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq" event={"ID":"472ad38a-7bd8-44e3-87ce-9ef3a84ee342","Type":"ContainerDied","Data":"9ba455f41923ca15a5618299481ba615a0bd057abbd0542e7e50b99c35fe20e7"}
Nov 03 21:19:49.909575 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.909563    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"
Nov 03 21:19:49.909894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:49.909595    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ba455f41923ca15a5618299481ba615a0bd057abbd0542e7e50b99c35fe20e7"
Nov 03 21:19:52.881580 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:52.881546    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:56.042735 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:56.042699    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"]
Nov 03 21:19:56.089645 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:56.089610    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570bedb-mcd9j"]
Nov 03 21:19:56.108288 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:56.108257    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0dad2f-a03a-452b-8fba-4a20b2905abf" path="/var/lib/kubelet/pods/1b0dad2f-a03a-452b-8fba-4a20b2905abf/volumes"
Nov 03 21:19:58.448653 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:19:58.448620    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:20:12.770389 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:12.770347    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:20:18.484607 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:20:18.484565    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:20:20.414890 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.414841    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"]
Nov 03 21:20:20.416198 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.416167    5196 cpu_manager.go:414] "containerMap: removing container" podUID="472ad38a-7bd8-44e3-87ce-9ef3a84ee342" containerName="revoke-keys"
Nov 03 21:20:20.416364 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.416349    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472ad38a-7bd8-44e3-87ce-9ef3a84ee342" containerName="revoke-keys"
Nov 03 21:20:20.416625 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.416612    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="472ad38a-7bd8-44e3-87ce-9ef3a84ee342" containerName="revoke-keys"
Nov 03 21:20:20.420678 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.420654    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.470525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.470491    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"]
Nov 03 21:20:20.473313 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.473281    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5v4k\" (UniqueName: \"kubernetes.io/projected/09b092c1-efc2-4735-8cfe-b81b0182d868-kube-api-access-l5v4k\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.473451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.473431    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.473542 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.473481    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.498469 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.498439    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:20:20.514056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.514008    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:20:20.574773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.574736    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5v4k\" (UniqueName: \"kubernetes.io/projected/09b092c1-efc2-4735-8cfe-b81b0182d868-kube-api-access-l5v4k\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.574951 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.574845    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.574951 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.574882    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.575573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.575546    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.577502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.577472    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.628912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.628881    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5v4k\" (UniqueName: \"kubernetes.io/projected/09b092c1-efc2-4735-8cfe-b81b0182d868-kube-api-access-l5v4k\") pod \"maas-api-revoke-sub-2e1f5b5f-5dwc8\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:20.733750 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:20.733668    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:21.109914 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:21.109877    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"]
Nov 03 21:20:21.117474 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:20:21.117438    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b092c1_efc2_4735_8cfe_b81b0182d868.slice/crio-21af0604af9b1436a2bcebef97229672132d66d8142eabcff288e23df9b4a950 WatchSource:0}: Error finding container 21af0604af9b1436a2bcebef97229672132d66d8142eabcff288e23df9b4a950: Status 404 returned error can't find the container with id 21af0604af9b1436a2bcebef97229672132d66d8142eabcff288e23df9b4a950
Nov 03 21:20:21.146087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:21.146055    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8" event={"ID":"09b092c1-efc2-4735-8cfe-b81b0182d868","Type":"ContainerStarted","Data":"21af0604af9b1436a2bcebef97229672132d66d8142eabcff288e23df9b4a950"}
Nov 03 21:20:22.163340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:22.163302    5196 generic.go:356] "Generic (PLEG): container finished" podID="09b092c1-efc2-4735-8cfe-b81b0182d868" containerID="e5f5a6a60e6725a0afb8a2912bfcbac330f2f0e639fe72f7d295def1823404be" exitCode=0
Nov 03 21:20:22.163758 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:22.163392    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8" event={"ID":"09b092c1-efc2-4735-8cfe-b81b0182d868","Type":"ContainerDied","Data":"e5f5a6a60e6725a0afb8a2912bfcbac330f2f0e639fe72f7d295def1823404be"}
Nov 03 21:20:23.324041 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.324013    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:23.404157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.404122    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-xks-ca\") pod \"09b092c1-efc2-4735-8cfe-b81b0182d868\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") "
Nov 03 21:20:23.404359 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.404194    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b092c1-efc2-4735-8cfe-b81b0182d868-kube-api-access-l5v4k\" (UniqueName: \"kubernetes.io/projected/09b092c1-efc2-4735-8cfe-b81b0182d868-kube-api-access-l5v4k\") pod \"09b092c1-efc2-4735-8cfe-b81b0182d868\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") "
Nov 03 21:20:23.404359 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.404350    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-openshift-ca\") pod \"09b092c1-efc2-4735-8cfe-b81b0182d868\" (UID: \"09b092c1-efc2-4735-8cfe-b81b0182d868\") "
Nov 03 21:20:23.404694 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.404659    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-openshift-ca" pod "09b092c1-efc2-4735-8cfe-b81b0182d868" (UID: "09b092c1-efc2-4735-8cfe-b81b0182d868"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:23.406441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.406419    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-xks-ca" pod "09b092c1-efc2-4735-8cfe-b81b0182d868" (UID: "09b092c1-efc2-4735-8cfe-b81b0182d868"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:20:23.406543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.406494    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b092c1-efc2-4735-8cfe-b81b0182d868-kube-api-access-l5v4k" pod "09b092c1-efc2-4735-8cfe-b81b0182d868" (UID: "09b092c1-efc2-4735-8cfe-b81b0182d868"). InnerVolumeSpecName "kube-api-access-l5v4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:23.506161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.506046    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:23.506161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.506080    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b092c1-efc2-4735-8cfe-b81b0182d868-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:23.506161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:23.506104    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5v4k\" (UniqueName: \"kubernetes.io/projected/09b092c1-efc2-4735-8cfe-b81b0182d868-kube-api-access-l5v4k\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:24.178153 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:24.178121    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"
Nov 03 21:20:24.178153 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:24.178131    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8" event={"ID":"09b092c1-efc2-4735-8cfe-b81b0182d868","Type":"ContainerDied","Data":"21af0604af9b1436a2bcebef97229672132d66d8142eabcff288e23df9b4a950"}
Nov 03 21:20:24.178345 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:24.178189    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21af0604af9b1436a2bcebef97229672132d66d8142eabcff288e23df9b4a950"
Nov 03 21:20:28.138131 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:28.138078    5196 scope.go:122] "RemoveContainer" containerID="c948538ca077cf533c430898f5e88e64f9524dec1d017f89240c3b45c1155c6b"
Nov 03 21:20:28.148083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:28.148062    5196 scope.go:122] "RemoveContainer" containerID="49c2e2f5ae94d4004e5a54da3e533775e23072efd9aff7a1e59cede9ddd23270"
Nov 03 21:20:34.787700 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:34.787649    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:20:42.429605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.429566    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"]
Nov 03 21:20:42.430384 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.430361    5196 cpu_manager.go:414] "containerMap: removing container" podUID="09b092c1-efc2-4735-8cfe-b81b0182d868" containerName="revoke-keys"
Nov 03 21:20:42.430498 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.430386    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b092c1-efc2-4735-8cfe-b81b0182d868" containerName="revoke-keys"
Nov 03 21:20:42.430559 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.430513    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b092c1-efc2-4735-8cfe-b81b0182d868" containerName="revoke-keys"
Nov 03 21:20:42.436240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.436206    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.485028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.484973    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"]
Nov 03 21:20:42.527690 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.527657    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:20:42.527894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.527716    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:20:42.541448 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.541412    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hgn\" (UniqueName: \"kubernetes.io/projected/21ff8a31-394e-432a-b498-ca3933d657ee-kube-api-access-x7hgn\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.541595 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.541550    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.541670 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.541640    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.642618 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.642580    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hgn\" (UniqueName: \"kubernetes.io/projected/21ff8a31-394e-432a-b498-ca3933d657ee-kube-api-access-x7hgn\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.642828 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.642665    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.642828 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.642725    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.643497 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.643475    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.645242 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.645224    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.696663 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.696588    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hgn\" (UniqueName: \"kubernetes.io/projected/21ff8a31-394e-432a-b498-ca3933d657ee-kube-api-access-x7hgn\") pod \"maas-api-revoke-sub-ec843018-m8wk5\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:42.755326 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:42.755279    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:43.136459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:43.136426    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"]
Nov 03 21:20:43.143456 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:20:43.143417    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ff8a31_394e_432a_b498_ca3933d657ee.slice/crio-6cd012000ba6ddadfdf18ecca08c87f187a0c02076ab17afe9368b0ac6583c41 WatchSource:0}: Error finding container 6cd012000ba6ddadfdf18ecca08c87f187a0c02076ab17afe9368b0ac6583c41: Status 404 returned error can't find the container with id 6cd012000ba6ddadfdf18ecca08c87f187a0c02076ab17afe9368b0ac6583c41
Nov 03 21:20:43.340061 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:43.340027    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5" event={"ID":"21ff8a31-394e-432a-b498-ca3933d657ee","Type":"ContainerStarted","Data":"6cd012000ba6ddadfdf18ecca08c87f187a0c02076ab17afe9368b0ac6583c41"}
Nov 03 21:20:44.348529 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:44.348492    5196 generic.go:356] "Generic (PLEG): container finished" podID="21ff8a31-394e-432a-b498-ca3933d657ee" containerID="be3d9c4afbca5d6ee04cbb5210deacf95305605a35da16aa18fd13cec604b880" exitCode=0
Nov 03 21:20:44.348971 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:44.348580    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5" event={"ID":"21ff8a31-394e-432a-b498-ca3933d657ee","Type":"ContainerDied","Data":"be3d9c4afbca5d6ee04cbb5210deacf95305605a35da16aa18fd13cec604b880"}
Nov 03 21:20:45.504078 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.504047    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:45.583033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.583000    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-openshift-ca\") pod \"21ff8a31-394e-432a-b498-ca3933d657ee\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") "
Nov 03 21:20:45.583240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.583054    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-xks-ca\") pod \"21ff8a31-394e-432a-b498-ca3933d657ee\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") "
Nov 03 21:20:45.583240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.583141    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ff8a31-394e-432a-b498-ca3933d657ee-kube-api-access-x7hgn\" (UniqueName: \"kubernetes.io/projected/21ff8a31-394e-432a-b498-ca3933d657ee-kube-api-access-x7hgn\") pod \"21ff8a31-394e-432a-b498-ca3933d657ee\" (UID: \"21ff8a31-394e-432a-b498-ca3933d657ee\") "
Nov 03 21:20:45.583437 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.583413    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-openshift-ca" pod "21ff8a31-394e-432a-b498-ca3933d657ee" (UID: "21ff8a31-394e-432a-b498-ca3933d657ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:45.585542 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.585512    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-xks-ca" pod "21ff8a31-394e-432a-b498-ca3933d657ee" (UID: "21ff8a31-394e-432a-b498-ca3933d657ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:20:45.585673 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.585630    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ff8a31-394e-432a-b498-ca3933d657ee-kube-api-access-x7hgn" pod "21ff8a31-394e-432a-b498-ca3933d657ee" (UID: "21ff8a31-394e-432a-b498-ca3933d657ee"). InnerVolumeSpecName "kube-api-access-x7hgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:45.684756 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.684653    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hgn\" (UniqueName: \"kubernetes.io/projected/21ff8a31-394e-432a-b498-ca3933d657ee-kube-api-access-x7hgn\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:45.684756 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.684699    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:45.684756 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:45.684711    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ff8a31-394e-432a-b498-ca3933d657ee-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:46.363389 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:46.363361    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"
Nov 03 21:20:46.363581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:46.363360    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5" event={"ID":"21ff8a31-394e-432a-b498-ca3933d657ee","Type":"ContainerDied","Data":"6cd012000ba6ddadfdf18ecca08c87f187a0c02076ab17afe9368b0ac6583c41"}
Nov 03 21:20:46.363581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:46.363478    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd012000ba6ddadfdf18ecca08c87f187a0c02076ab17afe9368b0ac6583c41"
Nov 03 21:20:48.484458 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:20:48.484423    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:20:53.132678 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:53.132637    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:53.325749 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:53.325710    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:55.094533 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:55.094492    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:20:59.377532 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:20:59.377487    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:11.557853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:11.557816    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:21:18.484581 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:21:18.484538    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:21:28.178592 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:28.178558    5196 scope.go:122] "RemoveContainer" containerID="bfe6b47675a6efdab711a10f8dd1127ab5de5fcb3b301149c1724a875be02624"
Nov 03 21:21:28.189046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:28.189026    5196 scope.go:122] "RemoveContainer" containerID="5432978341b76514d9790a840613dcbb4193fb3c195c8ce5826dc2f66f595250"
Nov 03 21:21:28.199051 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:28.199030    5196 scope.go:122] "RemoveContainer" containerID="2cd1dcaf42ce8b3bbe5f8c2080b48cb0676404977170ce15156b30aa632a6e1f"
Nov 03 21:21:28.208453 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:28.208430    5196 scope.go:122] "RemoveContainer" containerID="2196aee6fc13721ac4176bcb169156acc8aef02c786f39545e6790b9d7aec191"
Nov 03 21:21:29.951804 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:29.951772    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"]
Nov 03 21:21:29.952314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:29.952300    5196 cpu_manager.go:414] "containerMap: removing container" podUID="21ff8a31-394e-432a-b498-ca3933d657ee" containerName="revoke-keys"
Nov 03 21:21:29.952367 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:29.952315    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ff8a31-394e-432a-b498-ca3933d657ee" containerName="revoke-keys"
Nov 03 21:21:29.952423 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:29.952414    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ff8a31-394e-432a-b498-ca3933d657ee" containerName="revoke-keys"
Nov 03 21:21:29.957385 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:29.957360    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.009242 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.009208    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"]
Nov 03 21:21:30.052365 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.052332    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:21:30.052561 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.052439    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:21:30.076904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.076859    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7xx\" (UniqueName: \"kubernetes.io/projected/036e543d-0cbf-4a4c-b978-fe8dbaca410b-kube-api-access-nc7xx\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.076904 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.076907    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.077183 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.077154    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.178785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.178742    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.178993 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.178829    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7xx\" (UniqueName: \"kubernetes.io/projected/036e543d-0cbf-4a4c-b978-fe8dbaca410b-kube-api-access-nc7xx\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.178993 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.178859    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.179556 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.179531    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.181349 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.181330    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.233409 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.233332    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7xx\" (UniqueName: \"kubernetes.io/projected/036e543d-0cbf-4a4c-b978-fe8dbaca410b-kube-api-access-nc7xx\") pod \"maas-api-revoke-sub-046e1343-fcwbr\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.271921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.271878    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:30.648043 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.648007    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"]
Nov 03 21:21:30.654742 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:21:30.654706    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036e543d_0cbf_4a4c_b978_fe8dbaca410b.slice/crio-a4529a61a8ed3c7ad58c564c88e07988d064a33c70f665a7cdeaada135ae06c0 WatchSource:0}: Error finding container a4529a61a8ed3c7ad58c564c88e07988d064a33c70f665a7cdeaada135ae06c0: Status 404 returned error can't find the container with id a4529a61a8ed3c7ad58c564c88e07988d064a33c70f665a7cdeaada135ae06c0
Nov 03 21:21:30.706981 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:30.706942    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr" event={"ID":"036e543d-0cbf-4a4c-b978-fe8dbaca410b","Type":"ContainerStarted","Data":"a4529a61a8ed3c7ad58c564c88e07988d064a33c70f665a7cdeaada135ae06c0"}
Nov 03 21:21:31.715284 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:31.715249    5196 generic.go:356] "Generic (PLEG): container finished" podID="036e543d-0cbf-4a4c-b978-fe8dbaca410b" containerID="5b0a2d02b97079dd371d1e0430184363a32b3e61c75f8e07c1bf88c0cc9cb361" exitCode=0
Nov 03 21:21:31.715665 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:31.715387    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr" event={"ID":"036e543d-0cbf-4a4c-b978-fe8dbaca410b","Type":"ContainerDied","Data":"5b0a2d02b97079dd371d1e0430184363a32b3e61c75f8e07c1bf88c0cc9cb361"}
Nov 03 21:21:32.880184 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:32.880157    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:33.011650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.011564    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-openshift-ca\") pod \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") "
Nov 03 21:21:33.011813 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.011708    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036e543d-0cbf-4a4c-b978-fe8dbaca410b-kube-api-access-nc7xx\" (UniqueName: \"kubernetes.io/projected/036e543d-0cbf-4a4c-b978-fe8dbaca410b-kube-api-access-nc7xx\") pod \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") "
Nov 03 21:21:33.011813 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.011740    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-xks-ca\") pod \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\" (UID: \"036e543d-0cbf-4a4c-b978-fe8dbaca410b\") "
Nov 03 21:21:33.011990 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.011962    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-openshift-ca" pod "036e543d-0cbf-4a4c-b978-fe8dbaca410b" (UID: "036e543d-0cbf-4a4c-b978-fe8dbaca410b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:33.012145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.012113    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:33.014009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.013980    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-xks-ca" pod "036e543d-0cbf-4a4c-b978-fe8dbaca410b" (UID: "036e543d-0cbf-4a4c-b978-fe8dbaca410b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:33.014150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.014052    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036e543d-0cbf-4a4c-b978-fe8dbaca410b-kube-api-access-nc7xx" pod "036e543d-0cbf-4a4c-b978-fe8dbaca410b" (UID: "036e543d-0cbf-4a4c-b978-fe8dbaca410b"). InnerVolumeSpecName "kube-api-access-nc7xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:33.112740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.112705    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc7xx\" (UniqueName: \"kubernetes.io/projected/036e543d-0cbf-4a4c-b978-fe8dbaca410b-kube-api-access-nc7xx\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:33.112740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.112734    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/036e543d-0cbf-4a4c-b978-fe8dbaca410b-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:33.729178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.729147    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"
Nov 03 21:21:33.729178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.729161    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr" event={"ID":"036e543d-0cbf-4a4c-b978-fe8dbaca410b","Type":"ContainerDied","Data":"a4529a61a8ed3c7ad58c564c88e07988d064a33c70f665a7cdeaada135ae06c0"}
Nov 03 21:21:33.729447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:33.729219    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4529a61a8ed3c7ad58c564c88e07988d064a33c70f665a7cdeaada135ae06c0"
Nov 03 21:21:40.697081 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.697039    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"]
Nov 03 21:21:40.697707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.697690    5196 cpu_manager.go:414] "containerMap: removing container" podUID="036e543d-0cbf-4a4c-b978-fe8dbaca410b" containerName="revoke-keys"
Nov 03 21:21:40.697758 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.697709    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036e543d-0cbf-4a4c-b978-fe8dbaca410b" containerName="revoke-keys"
Nov 03 21:21:40.697795 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.697789    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="036e543d-0cbf-4a4c-b978-fe8dbaca410b" containerName="revoke-keys"
Nov 03 21:21:40.702128 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.702077    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.754894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.754855    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"]
Nov 03 21:21:40.789842 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.789806    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pc5\" (UniqueName: \"kubernetes.io/projected/dc73368c-c20c-4b05-bf52-3020a1bac6ae-kube-api-access-b4pc5\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.790040 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.789856    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.790040 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.789971    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.796526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.796496    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:21:40.796665 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.796551    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:21:40.890875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.890838    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.891086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.890935    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.891086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.891029    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pc5\" (UniqueName: \"kubernetes.io/projected/dc73368c-c20c-4b05-bf52-3020a1bac6ae-kube-api-access-b4pc5\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.891565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.891527    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.893680 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.893659    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:40.945839 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:40.945811    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pc5\" (UniqueName: \"kubernetes.io/projected/dc73368c-c20c-4b05-bf52-3020a1bac6ae-kube-api-access-b4pc5\") pod \"maas-api-revoke-sub-d3fba056-vftxl\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:41.016598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:41.016501    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:41.392306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:41.392270    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"]
Nov 03 21:21:41.399669 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:21:41.399634    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc73368c_c20c_4b05_bf52_3020a1bac6ae.slice/crio-cdc8776e01928bd45876cc133c3d8c088b85d1cc6f07693332505fa1a88de0ee WatchSource:0}: Error finding container cdc8776e01928bd45876cc133c3d8c088b85d1cc6f07693332505fa1a88de0ee: Status 404 returned error can't find the container with id cdc8776e01928bd45876cc133c3d8c088b85d1cc6f07693332505fa1a88de0ee
Nov 03 21:21:41.787245 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:41.787205    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl" event={"ID":"dc73368c-c20c-4b05-bf52-3020a1bac6ae","Type":"ContainerStarted","Data":"6bebdf4cafb03ed485d2bfd6736e527af0afc76525a18f0483134f0ec884c1ab"}
Nov 03 21:21:41.787701 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:41.787272    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl" event={"ID":"dc73368c-c20c-4b05-bf52-3020a1bac6ae","Type":"ContainerStarted","Data":"cdc8776e01928bd45876cc133c3d8c088b85d1cc6f07693332505fa1a88de0ee"}
Nov 03 21:21:41.896154 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:41.896042    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl" podStartSLOduration=1.896026011 podStartE2EDuration="1.896026011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:41.893624983 +0000 UTC m=+3141.897583712" watchObservedRunningTime="2026-11-03 21:21:41.896026011 +0000 UTC m=+3141.899984787"
Nov 03 21:21:42.679439 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:42.679402    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:21:42.796305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:42.796258    5196 generic.go:356] "Generic (PLEG): container finished" podID="dc73368c-c20c-4b05-bf52-3020a1bac6ae" containerID="6bebdf4cafb03ed485d2bfd6736e527af0afc76525a18f0483134f0ec884c1ab" exitCode=0
Nov 03 21:21:42.796712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:42.796340    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl" event={"ID":"dc73368c-c20c-4b05-bf52-3020a1bac6ae","Type":"ContainerDied","Data":"6bebdf4cafb03ed485d2bfd6736e527af0afc76525a18f0483134f0ec884c1ab"}
Nov 03 21:21:43.745145 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:43.745075    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:43.977243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:43.977141    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:44.119552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.119529    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:44.313206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.265580    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-xks-ca\") pod \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") "
Nov 03 21:21:44.313206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.265676    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc73368c-c20c-4b05-bf52-3020a1bac6ae-kube-api-access-b4pc5\" (UniqueName: \"kubernetes.io/projected/dc73368c-c20c-4b05-bf52-3020a1bac6ae-kube-api-access-b4pc5\") pod \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") "
Nov 03 21:21:44.313206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.265710    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-openshift-ca\") pod \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\" (UID: \"dc73368c-c20c-4b05-bf52-3020a1bac6ae\") "
Nov 03 21:21:44.313206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.290341    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-openshift-ca" pod "dc73368c-c20c-4b05-bf52-3020a1bac6ae" (UID: "dc73368c-c20c-4b05-bf52-3020a1bac6ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:44.321428 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.321364    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-xks-ca" pod "dc73368c-c20c-4b05-bf52-3020a1bac6ae" (UID: "dc73368c-c20c-4b05-bf52-3020a1bac6ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:44.321972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.321940    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc73368c-c20c-4b05-bf52-3020a1bac6ae-kube-api-access-b4pc5" pod "dc73368c-c20c-4b05-bf52-3020a1bac6ae" (UID: "dc73368c-c20c-4b05-bf52-3020a1bac6ae"). InnerVolumeSpecName "kube-api-access-b4pc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:44.374121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.367994    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pc5\" (UniqueName: \"kubernetes.io/projected/dc73368c-c20c-4b05-bf52-3020a1bac6ae-kube-api-access-b4pc5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:44.374456 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.374423    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:44.397712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.397674    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc73368c-c20c-4b05-bf52-3020a1bac6ae-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:44.819132 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.819076    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"
Nov 03 21:21:44.819132 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.819086    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl" event={"ID":"dc73368c-c20c-4b05-bf52-3020a1bac6ae","Type":"ContainerDied","Data":"cdc8776e01928bd45876cc133c3d8c088b85d1cc6f07693332505fa1a88de0ee"}
Nov 03 21:21:44.819432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:44.819174    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdc8776e01928bd45876cc133c3d8c088b85d1cc6f07693332505fa1a88de0ee"
Nov 03 21:21:48.486568 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:21:48.486528    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:21:53.380929 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:53.380885    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:21:53.502630 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:21:53.502589    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:01.073182 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:01.067312    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"]
Nov 03 21:22:01.113216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:01.113166    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"]
Nov 03 21:22:01.115720 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:01.115591    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24402417-9bx9j"]
Nov 03 21:22:01.120446 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:01.120418    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a9432de-vwm7j"]
Nov 03 21:22:02.114084 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:02.114046    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2287ee7-cce8-4670-907a-7b373f80382b" path="/var/lib/kubelet/pods/a2287ee7-cce8-4670-907a-7b373f80382b/volumes"
Nov 03 21:22:02.114571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:02.114552    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5497d40-7d72-46b9-9571-239251bb3ac9" path="/var/lib/kubelet/pods/f5497d40-7d72-46b9-9571-239251bb3ac9/volumes"
Nov 03 21:22:02.449349 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:02.449245    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:05.266030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:05.265991    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:22:15.549667 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.549629    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"]
Nov 03 21:22:15.550186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.550171    5196 cpu_manager.go:414] "containerMap: removing container" podUID="dc73368c-c20c-4b05-bf52-3020a1bac6ae" containerName="revoke-keys"
Nov 03 21:22:15.550239 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.550188    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc73368c-c20c-4b05-bf52-3020a1bac6ae" containerName="revoke-keys"
Nov 03 21:22:15.550302 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.550293    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc73368c-c20c-4b05-bf52-3020a1bac6ae" containerName="revoke-keys"
Nov 03 21:22:15.556227 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.556193    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.609153 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.609123    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"]
Nov 03 21:22:15.652341 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.652308    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:22:15.652519 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.652379    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:22:15.666030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.665993    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.666030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.666030    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn96c\" (UniqueName: \"kubernetes.io/projected/a51a5500-420c-49b0-a25e-e81c369ca617-kube-api-access-zn96c\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.666301 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.666198    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.767477 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.767438    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn96c\" (UniqueName: \"kubernetes.io/projected/a51a5500-420c-49b0-a25e-e81c369ca617-kube-api-access-zn96c\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.767659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.767556    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.767659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.767614    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.768222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.768198    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.770322 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.770299    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.821535 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.821506    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn96c\" (UniqueName: \"kubernetes.io/projected/a51a5500-420c-49b0-a25e-e81c369ca617-kube-api-access-zn96c\") pod \"maas-api-revoke-sub-ae594979-vtngv\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:15.873740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:15.873709    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:16.248149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:16.248044    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"]
Nov 03 21:22:16.254786 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:22:16.254756    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51a5500_420c_49b0_a25e_e81c369ca617.slice/crio-ebde840232629102caf44efa84ebca4714e1abf05e60bfaf2571a2458d750ddb WatchSource:0}: Error finding container ebde840232629102caf44efa84ebca4714e1abf05e60bfaf2571a2458d750ddb: Status 404 returned error can't find the container with id ebde840232629102caf44efa84ebca4714e1abf05e60bfaf2571a2458d750ddb
Nov 03 21:22:17.107417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:17.107380    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv" event={"ID":"a51a5500-420c-49b0-a25e-e81c369ca617","Type":"ContainerStarted","Data":"05bd65ef8eea75f3eeae9b50eb7137d8b3dbfc9e27684170404c3b7250402230"}
Nov 03 21:22:17.107813 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:17.107447    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv" event={"ID":"a51a5500-420c-49b0-a25e-e81c369ca617","Type":"ContainerStarted","Data":"ebde840232629102caf44efa84ebca4714e1abf05e60bfaf2571a2458d750ddb"}
Nov 03 21:22:17.218304 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:17.218217    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv" podStartSLOduration=2.21819752 podStartE2EDuration="2.21819752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:17.214503786 +0000 UTC m=+3177.218462527" watchObservedRunningTime="2026-11-03 21:22:17.21819752 +0000 UTC m=+3177.222156248"
Nov 03 21:22:18.114611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:18.114578    5196 generic.go:356] "Generic (PLEG): container finished" podID="a51a5500-420c-49b0-a25e-e81c369ca617" containerID="05bd65ef8eea75f3eeae9b50eb7137d8b3dbfc9e27684170404c3b7250402230" exitCode=0
Nov 03 21:22:18.115126 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:18.114659    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv" event={"ID":"a51a5500-420c-49b0-a25e-e81c369ca617","Type":"ContainerDied","Data":"05bd65ef8eea75f3eeae9b50eb7137d8b3dbfc9e27684170404c3b7250402230"}
Nov 03 21:22:18.485659 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:22:18.485569    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:22:19.282620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.282586    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:19.406046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.405961    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-xks-ca\") pod \"a51a5500-420c-49b0-a25e-e81c369ca617\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") "
Nov 03 21:22:19.406046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.406026    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51a5500-420c-49b0-a25e-e81c369ca617-kube-api-access-zn96c\" (UniqueName: \"kubernetes.io/projected/a51a5500-420c-49b0-a25e-e81c369ca617-kube-api-access-zn96c\") pod \"a51a5500-420c-49b0-a25e-e81c369ca617\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") "
Nov 03 21:22:19.406354 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.406177    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-openshift-ca\") pod \"a51a5500-420c-49b0-a25e-e81c369ca617\" (UID: \"a51a5500-420c-49b0-a25e-e81c369ca617\") "
Nov 03 21:22:19.406537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.406508    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-openshift-ca" pod "a51a5500-420c-49b0-a25e-e81c369ca617" (UID: "a51a5500-420c-49b0-a25e-e81c369ca617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:19.406680 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.406585    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.408970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.408942    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-xks-ca" pod "a51a5500-420c-49b0-a25e-e81c369ca617" (UID: "a51a5500-420c-49b0-a25e-e81c369ca617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:22:19.409081 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.408968    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51a5500-420c-49b0-a25e-e81c369ca617-kube-api-access-zn96c" pod "a51a5500-420c-49b0-a25e-e81c369ca617" (UID: "a51a5500-420c-49b0-a25e-e81c369ca617"). InnerVolumeSpecName "kube-api-access-zn96c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:19.508165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.508080    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a51a5500-420c-49b0-a25e-e81c369ca617-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.508165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:19.508163    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn96c\" (UniqueName: \"kubernetes.io/projected/a51a5500-420c-49b0-a25e-e81c369ca617-kube-api-access-zn96c\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:20.138316 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:20.138282    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"
Nov 03 21:22:20.138316 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:20.138293    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv" event={"ID":"a51a5500-420c-49b0-a25e-e81c369ca617","Type":"ContainerDied","Data":"ebde840232629102caf44efa84ebca4714e1abf05e60bfaf2571a2458d750ddb"}
Nov 03 21:22:20.138586 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:20.138363    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebde840232629102caf44efa84ebca4714e1abf05e60bfaf2571a2458d750ddb"
Nov 03 21:22:28.248388 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:28.248355    5196 scope.go:122] "RemoveContainer" containerID="e341433f0d8c053114bdb110df5f7307eb3ce7aa43ea14bba80e6a3d98d222b0"
Nov 03 21:22:28.258507 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:28.258480    5196 scope.go:122] "RemoveContainer" containerID="5d81673e6f1c2a6b2547823e65170366106982af84c8b8bb720d95f3f8951fea"
Nov 03 21:22:28.286740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:28.286716    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:22:37.351677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:37.351626    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:22:43.335017 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.334983    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"]
Nov 03 21:22:43.335639 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.335621    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a51a5500-420c-49b0-a25e-e81c369ca617" containerName="revoke-keys"
Nov 03 21:22:43.335695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.335642    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51a5500-420c-49b0-a25e-e81c369ca617" containerName="revoke-keys"
Nov 03 21:22:43.335775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.335764    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51a5500-420c-49b0-a25e-e81c369ca617" containerName="revoke-keys"
Nov 03 21:22:43.339258 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.339233    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.392047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.392016    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"]
Nov 03 21:22:43.394775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.394723    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhh7b\" (UniqueName: \"kubernetes.io/projected/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-kube-api-access-zhh7b\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.394969 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.394860    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.394969 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.394954    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.434664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.434629    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:22:43.434860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.434841    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:22:43.496038 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.496000    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhh7b\" (UniqueName: \"kubernetes.io/projected/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-kube-api-access-zhh7b\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.496253 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.496052    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.496253 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.496177    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.496752 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.496722    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.498821 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.498801    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.550395 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.550365    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhh7b\" (UniqueName: \"kubernetes.io/projected/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-kube-api-access-zhh7b\") pod \"maas-api-revoke-sub-a4aef7d7-hbsdk\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:43.653617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:43.653533    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:44.026700 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:44.026612    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"]
Nov 03 21:22:44.033638 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:22:44.033610    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceabb2eb_9b79_4fe4_82db_b0e8465ac623.slice/crio-5bb79c5ed95f467e6d6f5a65467b3e2c87bfeb711c3bdc8d70253041b41af27f WatchSource:0}: Error finding container 5bb79c5ed95f467e6d6f5a65467b3e2c87bfeb711c3bdc8d70253041b41af27f: Status 404 returned error can't find the container with id 5bb79c5ed95f467e6d6f5a65467b3e2c87bfeb711c3bdc8d70253041b41af27f
Nov 03 21:22:44.348515 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:44.348443    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk" event={"ID":"ceabb2eb-9b79-4fe4-82db-b0e8465ac623","Type":"ContainerStarted","Data":"5bb79c5ed95f467e6d6f5a65467b3e2c87bfeb711c3bdc8d70253041b41af27f"}
Nov 03 21:22:45.356806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:45.356771    5196 generic.go:356] "Generic (PLEG): container finished" podID="ceabb2eb-9b79-4fe4-82db-b0e8465ac623" containerID="072613f7bed19a9e243dd95019bdd69565dc364a8eb40d425cf273c6db54aca2" exitCode=0
Nov 03 21:22:45.357257 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:45.356827    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk" event={"ID":"ceabb2eb-9b79-4fe4-82db-b0e8465ac623","Type":"ContainerDied","Data":"072613f7bed19a9e243dd95019bdd69565dc364a8eb40d425cf273c6db54aca2"}
Nov 03 21:22:46.518542 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.518517    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:46.632222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.632082    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-kube-api-access-zhh7b\" (UniqueName: \"kubernetes.io/projected/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-kube-api-access-zhh7b\") pod \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") "
Nov 03 21:22:46.632434 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.632233    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-xks-ca\") pod \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") "
Nov 03 21:22:46.632434 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.632351    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-openshift-ca\") pod \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\" (UID: \"ceabb2eb-9b79-4fe4-82db-b0e8465ac623\") "
Nov 03 21:22:46.633069 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.632926    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-openshift-ca" pod "ceabb2eb-9b79-4fe4-82db-b0e8465ac623" (UID: "ceabb2eb-9b79-4fe4-82db-b0e8465ac623"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:46.634747 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.634711    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-xks-ca" pod "ceabb2eb-9b79-4fe4-82db-b0e8465ac623" (UID: "ceabb2eb-9b79-4fe4-82db-b0e8465ac623"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:22:46.635234 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.635205    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-kube-api-access-zhh7b" pod "ceabb2eb-9b79-4fe4-82db-b0e8465ac623" (UID: "ceabb2eb-9b79-4fe4-82db-b0e8465ac623"). InnerVolumeSpecName "kube-api-access-zhh7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:46.733444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.733404    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhh7b\" (UniqueName: \"kubernetes.io/projected/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-kube-api-access-zhh7b\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:46.733444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.733441    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:46.733444 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:46.733452    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceabb2eb-9b79-4fe4-82db-b0e8465ac623-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:47.370827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:47.370796    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"
Nov 03 21:22:47.371020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:47.370835    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk" event={"ID":"ceabb2eb-9b79-4fe4-82db-b0e8465ac623","Type":"ContainerDied","Data":"5bb79c5ed95f467e6d6f5a65467b3e2c87bfeb711c3bdc8d70253041b41af27f"}
Nov 03 21:22:47.371020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:47.370889    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bb79c5ed95f467e6d6f5a65467b3e2c87bfeb711c3bdc8d70253041b41af27f"
Nov 03 21:22:48.049111 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:48.049057    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"]
Nov 03 21:22:48.096361 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:48.096324    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6996b236-qmgmh"]
Nov 03 21:22:48.108577 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:48.108544    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78afa92b-29ba-4a52-be75-8c7125ca7c7d" path="/var/lib/kubelet/pods/78afa92b-29ba-4a52-be75-8c7125ca7c7d/volumes"
Nov 03 21:22:48.486586 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:22:48.486549    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:22:53.806982 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:53.806936    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:55.963430 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:22:55.963382    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:23:07.770134 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:07.770086    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:23:13.823157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.823122    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"]
Nov 03 21:23:13.823664 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.823649    5196 cpu_manager.go:414] "containerMap: removing container" podUID="ceabb2eb-9b79-4fe4-82db-b0e8465ac623" containerName="revoke-keys"
Nov 03 21:23:13.823709 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.823666    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceabb2eb-9b79-4fe4-82db-b0e8465ac623" containerName="revoke-keys"
Nov 03 21:23:13.823785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.823776    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceabb2eb-9b79-4fe4-82db-b0e8465ac623" containerName="revoke-keys"
Nov 03 21:23:13.828544 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.828518    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:13.878371 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.878335    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"]
Nov 03 21:23:13.921740 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.921695    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:23:13.921939 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.921761    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:23:13.929380 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.929344    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:13.929512 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.929399    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:13.929565 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:13.929508    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvfwk\" (UniqueName: \"kubernetes.io/projected/a684792a-f808-4d8a-ba50-9e089dfe6194-kube-api-access-tvfwk\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:14.030903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.030862    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:14.030903 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.030908    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:14.031181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.030941    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvfwk\" (UniqueName: \"kubernetes.io/projected/a684792a-f808-4d8a-ba50-9e089dfe6194-kube-api-access-tvfwk\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:14.031702 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.031680    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:14.033603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.033584    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:14.084941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.084873    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvfwk\" (UniqueName: \"kubernetes.io/projected/a684792a-f808-4d8a-ba50-9e089dfe6194-kube-api-access-tvfwk\") pod \"maas-api-revoke-sub-79805ca9-sqdvb\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:14.142704 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.142669    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:14.520141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.520046    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"]
Nov 03 21:23:14.527065 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:23:14.527031    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda684792a_f808_4d8a_ba50_9e089dfe6194.slice/crio-f724c90deef1d178d7c3bd6a056922095be799e0e2a8d5cec037a3b908a6c4a9 WatchSource:0}: Error finding container f724c90deef1d178d7c3bd6a056922095be799e0e2a8d5cec037a3b908a6c4a9: Status 404 returned error can't find the container with id f724c90deef1d178d7c3bd6a056922095be799e0e2a8d5cec037a3b908a6c4a9
Nov 03 21:23:14.582330 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:14.582293    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb" event={"ID":"a684792a-f808-4d8a-ba50-9e089dfe6194","Type":"ContainerStarted","Data":"f724c90deef1d178d7c3bd6a056922095be799e0e2a8d5cec037a3b908a6c4a9"}
Nov 03 21:23:15.590403 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:15.590369    5196 generic.go:356] "Generic (PLEG): container finished" podID="a684792a-f808-4d8a-ba50-9e089dfe6194" containerID="0bc96037240d37973ef46b4ade0bf1493e023d53857d3a0cd8ded15f3a3912e7" exitCode=0
Nov 03 21:23:15.590819 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:15.590452    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb" event={"ID":"a684792a-f808-4d8a-ba50-9e089dfe6194","Type":"ContainerDied","Data":"0bc96037240d37973ef46b4ade0bf1493e023d53857d3a0cd8ded15f3a3912e7"}
Nov 03 21:23:16.754967 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.754944    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:16.865511 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.865429    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a684792a-f808-4d8a-ba50-9e089dfe6194-kube-api-access-tvfwk\" (UniqueName: \"kubernetes.io/projected/a684792a-f808-4d8a-ba50-9e089dfe6194-kube-api-access-tvfwk\") pod \"a684792a-f808-4d8a-ba50-9e089dfe6194\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") "
Nov 03 21:23:16.865511 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.865495    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-xks-ca\") pod \"a684792a-f808-4d8a-ba50-9e089dfe6194\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") "
Nov 03 21:23:16.865758 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.865555    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-openshift-ca\") pod \"a684792a-f808-4d8a-ba50-9e089dfe6194\" (UID: \"a684792a-f808-4d8a-ba50-9e089dfe6194\") "
Nov 03 21:23:16.865956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.865928    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-openshift-ca" pod "a684792a-f808-4d8a-ba50-9e089dfe6194" (UID: "a684792a-f808-4d8a-ba50-9e089dfe6194"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:23:16.867835 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.867807    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a684792a-f808-4d8a-ba50-9e089dfe6194-kube-api-access-tvfwk" pod "a684792a-f808-4d8a-ba50-9e089dfe6194" (UID: "a684792a-f808-4d8a-ba50-9e089dfe6194"). InnerVolumeSpecName "kube-api-access-tvfwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:16.867944 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.867809    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-xks-ca" pod "a684792a-f808-4d8a-ba50-9e089dfe6194" (UID: "a684792a-f808-4d8a-ba50-9e089dfe6194"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:23:16.966535 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.966489    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvfwk\" (UniqueName: \"kubernetes.io/projected/a684792a-f808-4d8a-ba50-9e089dfe6194-kube-api-access-tvfwk\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:16.966535 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.966529    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:16.966535 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:16.966539    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a684792a-f808-4d8a-ba50-9e089dfe6194-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:17.604203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:17.604162    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb" event={"ID":"a684792a-f808-4d8a-ba50-9e089dfe6194","Type":"ContainerDied","Data":"f724c90deef1d178d7c3bd6a056922095be799e0e2a8d5cec037a3b908a6c4a9"}
Nov 03 21:23:17.604203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:17.604193    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"
Nov 03 21:23:17.604432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:17.604220    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f724c90deef1d178d7c3bd6a056922095be799e0e2a8d5cec037a3b908a6c4a9"
Nov 03 21:23:18.485788 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:23:18.485751    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:23:27.902778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:27.902738    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:28.178793 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:28.178683    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:23:28.289899 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:28.289869    5196 scope.go:122] "RemoveContainer" containerID="ac498e729594284677d6da077a0615c972eca916418d145b88ca16c7c75f40e0"
Nov 03 21:23:28.300017 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:28.299995    5196 scope.go:122] "RemoveContainer" containerID="03f590365a930404c6df2f48981b739817014c124382aa85bd39182c271cc8e0"
Nov 03 21:23:28.309430 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:28.309405    5196 scope.go:122] "RemoveContainer" containerID="7451ab0be83a5d10c855d0b78ff61958d49b9a99db0f109a25216748adbd118e"
Nov 03 21:23:28.318223 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:28.318203    5196 scope.go:122] "RemoveContainer" containerID="36b3587492276131cf66ad59c25f859f795129cdcccbbd060695b55ccece71e5"
Nov 03 21:23:28.541256 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:28.541167    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:31.439571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:31.439525    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:31.633703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:31.633654    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:34.391597 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.391559    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"]
Nov 03 21:23:34.392157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.392141    5196 cpu_manager.go:414] "containerMap: removing container" podUID="a684792a-f808-4d8a-ba50-9e089dfe6194" containerName="revoke-keys"
Nov 03 21:23:34.392216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.392158    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a684792a-f808-4d8a-ba50-9e089dfe6194" containerName="revoke-keys"
Nov 03 21:23:34.392277 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.392267    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a684792a-f808-4d8a-ba50-9e089dfe6194" containerName="revoke-keys"
Nov 03 21:23:34.400872 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.400828    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.448285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.448243    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"]
Nov 03 21:23:34.461019 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.460979    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nwd\" (UniqueName: \"kubernetes.io/projected/645a0b25-344d-4567-91ba-1d95b0a24f9b-kube-api-access-z4nwd\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.461221 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.461169    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.461319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.461296    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.491040 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.491010    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:23:34.491248 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.491085    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:23:34.562582 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.562544    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nwd\" (UniqueName: \"kubernetes.io/projected/645a0b25-344d-4567-91ba-1d95b0a24f9b-kube-api-access-z4nwd\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.562798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.562616    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.562798 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.562658    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.563379 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.563338    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.565294 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.565267    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.616891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.616861    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nwd\" (UniqueName: \"kubernetes.io/projected/645a0b25-344d-4567-91ba-1d95b0a24f9b-kube-api-access-z4nwd\") pod \"maas-api-revoke-sub-0b7e8edb-78vfh\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:34.722886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:34.722800    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:35.108362 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:35.108329    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"]
Nov 03 21:23:35.116184 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:23:35.116154    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645a0b25_344d_4567_91ba_1d95b0a24f9b.slice/crio-dd39b6be2eff49324dfdb83748b58f1b84533df168963f357b9ee627a9f3860c WatchSource:0}: Error finding container dd39b6be2eff49324dfdb83748b58f1b84533df168963f357b9ee627a9f3860c: Status 404 returned error can't find the container with id dd39b6be2eff49324dfdb83748b58f1b84533df168963f357b9ee627a9f3860c
Nov 03 21:23:35.759677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:35.759586    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh" event={"ID":"645a0b25-344d-4567-91ba-1d95b0a24f9b","Type":"ContainerStarted","Data":"66dc5cef0754b02d8566e99b8c4b908222e77b4198d2e5af127e597212e65985"}
Nov 03 21:23:35.759677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:35.759646    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh" event={"ID":"645a0b25-344d-4567-91ba-1d95b0a24f9b","Type":"ContainerStarted","Data":"dd39b6be2eff49324dfdb83748b58f1b84533df168963f357b9ee627a9f3860c"}
Nov 03 21:23:35.868326 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:35.868269    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh" podStartSLOduration=1.868254484 podStartE2EDuration="1.868254484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:35.866271281 +0000 UTC m=+3255.870230023" watchObservedRunningTime="2026-11-03 21:23:35.868254484 +0000 UTC m=+3255.872213208"
Nov 03 21:23:36.767651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:36.767619    5196 generic.go:356] "Generic (PLEG): container finished" podID="645a0b25-344d-4567-91ba-1d95b0a24f9b" containerID="66dc5cef0754b02d8566e99b8c4b908222e77b4198d2e5af127e597212e65985" exitCode=0
Nov 03 21:23:36.768149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:36.767671    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh" event={"ID":"645a0b25-344d-4567-91ba-1d95b0a24f9b","Type":"ContainerDied","Data":"66dc5cef0754b02d8566e99b8c4b908222e77b4198d2e5af127e597212e65985"}
Nov 03 21:23:37.929025 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:37.928996    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:37.999887 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:37.999849    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-xks-ca\") pod \"645a0b25-344d-4567-91ba-1d95b0a24f9b\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") "
Nov 03 21:23:38.000066 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:37.999979    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645a0b25-344d-4567-91ba-1d95b0a24f9b-kube-api-access-z4nwd\" (UniqueName: \"kubernetes.io/projected/645a0b25-344d-4567-91ba-1d95b0a24f9b-kube-api-access-z4nwd\") pod \"645a0b25-344d-4567-91ba-1d95b0a24f9b\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") "
Nov 03 21:23:38.000066 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.000048    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-openshift-ca\") pod \"645a0b25-344d-4567-91ba-1d95b0a24f9b\" (UID: \"645a0b25-344d-4567-91ba-1d95b0a24f9b\") "
Nov 03 21:23:38.000455 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.000429    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-openshift-ca" pod "645a0b25-344d-4567-91ba-1d95b0a24f9b" (UID: "645a0b25-344d-4567-91ba-1d95b0a24f9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:23:38.002289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.002266    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-xks-ca" pod "645a0b25-344d-4567-91ba-1d95b0a24f9b" (UID: "645a0b25-344d-4567-91ba-1d95b0a24f9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:23:38.002441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.002419    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645a0b25-344d-4567-91ba-1d95b0a24f9b-kube-api-access-z4nwd" pod "645a0b25-344d-4567-91ba-1d95b0a24f9b" (UID: "645a0b25-344d-4567-91ba-1d95b0a24f9b"). InnerVolumeSpecName "kube-api-access-z4nwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:38.102024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.101980    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:38.102024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.102015    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/645a0b25-344d-4567-91ba-1d95b0a24f9b-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:38.102024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.102025    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nwd\" (UniqueName: \"kubernetes.io/projected/645a0b25-344d-4567-91ba-1d95b0a24f9b-kube-api-access-z4nwd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:38.781177 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.781137    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh" event={"ID":"645a0b25-344d-4567-91ba-1d95b0a24f9b","Type":"ContainerDied","Data":"dd39b6be2eff49324dfdb83748b58f1b84533df168963f357b9ee627a9f3860c"}
Nov 03 21:23:38.781372 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.781235    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd39b6be2eff49324dfdb83748b58f1b84533df168963f357b9ee627a9f3860c"
Nov 03 21:23:38.781372 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:38.781162    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"
Nov 03 21:23:45.680544 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.680501    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"]
Nov 03 21:23:45.681075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.681024    5196 cpu_manager.go:414] "containerMap: removing container" podUID="645a0b25-344d-4567-91ba-1d95b0a24f9b" containerName="revoke-keys"
Nov 03 21:23:45.681075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.681036    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645a0b25-344d-4567-91ba-1d95b0a24f9b" containerName="revoke-keys"
Nov 03 21:23:45.681207 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.681178    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="645a0b25-344d-4567-91ba-1d95b0a24f9b" containerName="revoke-keys"
Nov 03 21:23:45.684396 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.684375    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.738896 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.738855    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"]
Nov 03 21:23:45.781211 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.781172    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:23:45.781407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.781256    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:23:45.788231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.788189    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.788397 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.788372    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswzc\" (UniqueName: \"kubernetes.io/projected/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-kube-api-access-jswzc\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.788455 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.788407    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.889808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.889769    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswzc\" (UniqueName: \"kubernetes.io/projected/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-kube-api-access-jswzc\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.890028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.889833    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.890028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.889914    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.890564 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.890535    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.892448 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.892428    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.944403 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.944328    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswzc\" (UniqueName: \"kubernetes.io/projected/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-kube-api-access-jswzc\") pod \"maas-api-revoke-sub-3d4156f1-npwg4\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:45.998327 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:45.998283    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:46.374892 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:46.374859    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"]
Nov 03 21:23:46.381381 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:23:46.381350    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfdd80c_85a4_418b_9429_09d17ac8a1e5.slice/crio-05234be38b3d4434839f3d04bcb95bca1fe5738285f6b5414c6d15fb311b5665 WatchSource:0}: Error finding container 05234be38b3d4434839f3d04bcb95bca1fe5738285f6b5414c6d15fb311b5665: Status 404 returned error can't find the container with id 05234be38b3d4434839f3d04bcb95bca1fe5738285f6b5414c6d15fb311b5665
Nov 03 21:23:46.838975 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:46.838935    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4" event={"ID":"dbfdd80c-85a4-418b-9429-09d17ac8a1e5","Type":"ContainerStarted","Data":"88ff93008e1ea1dc39f46cd63aca76d87c0f51ae83dc9e633890b619326e611a"}
Nov 03 21:23:46.839558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:46.838999    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4" event={"ID":"dbfdd80c-85a4-418b-9429-09d17ac8a1e5","Type":"ContainerStarted","Data":"05234be38b3d4434839f3d04bcb95bca1fe5738285f6b5414c6d15fb311b5665"}
Nov 03 21:23:46.946325 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:46.946207    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4" podStartSLOduration=1.946192162 podStartE2EDuration="1.946192162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:46.945517593 +0000 UTC m=+3266.949476318" watchObservedRunningTime="2026-11-03 21:23:46.946192162 +0000 UTC m=+3266.950150879"
Nov 03 21:23:47.162133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:47.162059    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:23:47.235521 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:47.235488    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:47.894152 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:47.893184    5196 generic.go:356] "Generic (PLEG): container finished" podID="dbfdd80c-85a4-418b-9429-09d17ac8a1e5" containerID="88ff93008e1ea1dc39f46cd63aca76d87c0f51ae83dc9e633890b619326e611a" exitCode=0
Nov 03 21:23:47.894152 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:47.893390    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4" event={"ID":"dbfdd80c-85a4-418b-9429-09d17ac8a1e5","Type":"ContainerDied","Data":"88ff93008e1ea1dc39f46cd63aca76d87c0f51ae83dc9e633890b619326e611a"}
Nov 03 21:23:48.529763 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:23:48.529650    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:23:48.952249 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:48.952205    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-rrffz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:23:48.952873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:48.952836    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-rrffz" podUID="e67311d6-9cf7-4985-abb1-1cc9763e6156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:23:49.241293 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.240909    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:49.332062 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.332021    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-kube-api-access-jswzc\" (UniqueName: \"kubernetes.io/projected/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-kube-api-access-jswzc\") pod \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") "
Nov 03 21:23:49.332289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.332146    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-openshift-ca\") pod \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") "
Nov 03 21:23:49.332289 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.332183    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-xks-ca\") pod \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\" (UID: \"dbfdd80c-85a4-418b-9429-09d17ac8a1e5\") "
Nov 03 21:23:49.332520 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.332494    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-openshift-ca" pod "dbfdd80c-85a4-418b-9429-09d17ac8a1e5" (UID: "dbfdd80c-85a4-418b-9429-09d17ac8a1e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:23:49.332594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.332581    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:49.334460 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.334432    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-xks-ca" pod "dbfdd80c-85a4-418b-9429-09d17ac8a1e5" (UID: "dbfdd80c-85a4-418b-9429-09d17ac8a1e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:23:49.334544 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.334461    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-kube-api-access-jswzc" pod "dbfdd80c-85a4-418b-9429-09d17ac8a1e5" (UID: "dbfdd80c-85a4-418b-9429-09d17ac8a1e5"). InnerVolumeSpecName "kube-api-access-jswzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:49.433889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.433802    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:49.433889 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.433836    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswzc\" (UniqueName: \"kubernetes.io/projected/dbfdd80c-85a4-418b-9429-09d17ac8a1e5-kube-api-access-jswzc\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:49.923580 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.923542    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4" event={"ID":"dbfdd80c-85a4-418b-9429-09d17ac8a1e5","Type":"ContainerDied","Data":"05234be38b3d4434839f3d04bcb95bca1fe5738285f6b5414c6d15fb311b5665"}
Nov 03 21:23:49.923777 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.923602    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05234be38b3d4434839f3d04bcb95bca1fe5738285f6b5414c6d15fb311b5665"
Nov 03 21:23:49.923777 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:49.923554    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"
Nov 03 21:23:54.128702 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:54.128654    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:54.315581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:54.315520    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:54.463996 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:23:54.463892    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:24:00.049853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:00.049814    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:24:06.742069 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.742035    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"]
Nov 03 21:24:06.788887 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.788849    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-npwg4"]
Nov 03 21:24:06.855150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.855111    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"]
Nov 03 21:24:06.855703 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.855683    5196 cpu_manager.go:414] "containerMap: removing container" podUID="dbfdd80c-85a4-418b-9429-09d17ac8a1e5" containerName="revoke-keys"
Nov 03 21:24:06.855808 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.855705    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfdd80c-85a4-418b-9429-09d17ac8a1e5" containerName="revoke-keys"
Nov 03 21:24:06.855906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.855891    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfdd80c-85a4-418b-9429-09d17ac8a1e5" containerName="revoke-keys"
Nov 03 21:24:06.859523 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.859500    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:06.909607 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.909574    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:24:06.909607 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.909692    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:24:06.911605 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.911578    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"]
Nov 03 21:24:06.961443 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.961403    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849rw\" (UniqueName: \"kubernetes.io/projected/dafbb552-cee1-4efd-8209-a738b927ceec-kube-api-access-849rw\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:06.961658 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.961528    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:06.961658 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:06.961630    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:07.063141 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:07.063007    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849rw\" (UniqueName: \"kubernetes.io/projected/dafbb552-cee1-4efd-8209-a738b927ceec-kube-api-access-849rw\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:07.063346 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:07.063166    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:07.063346 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:07.063226    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:07.063906 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:07.063879    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:07.065745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:07.065724    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:07.117237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:07.117205    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-849rw\" (UniqueName: \"kubernetes.io/projected/dafbb552-cee1-4efd-8209-a738b927ceec-kube-api-access-849rw\") pod \"maas-api-revoke-sub-3d4156f1-r9mml\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:07.174117 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:07.174053    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:07.552069 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:07.552031    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"]
Nov 03 21:24:07.558813 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:24:07.558783    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafbb552_cee1_4efd_8209_a738b927ceec.slice/crio-8c48ade8ea251851e068a955026c18241c46050d0d99954278002f1486483ef1 WatchSource:0}: Error finding container 8c48ade8ea251851e068a955026c18241c46050d0d99954278002f1486483ef1: Status 404 returned error can't find the container with id 8c48ade8ea251851e068a955026c18241c46050d0d99954278002f1486483ef1
Nov 03 21:24:08.051314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:08.051267    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml" event={"ID":"dafbb552-cee1-4efd-8209-a738b927ceec","Type":"ContainerStarted","Data":"3f9f6059c9ea4af6d9e84744db39871457700f5e38b2eab5b9ac040031427422"}
Nov 03 21:24:08.051803 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:08.051358    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml" event={"ID":"dafbb552-cee1-4efd-8209-a738b927ceec","Type":"ContainerStarted","Data":"8c48ade8ea251851e068a955026c18241c46050d0d99954278002f1486483ef1"}
Nov 03 21:24:08.108299 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:08.108218    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbfdd80c-85a4-418b-9429-09d17ac8a1e5" path="/var/lib/kubelet/pods/dbfdd80c-85a4-418b-9429-09d17ac8a1e5/volumes"
Nov 03 21:24:08.160258 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:08.160207    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml" podStartSLOduration=2.160192831 podStartE2EDuration="2.160192831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:08.157755721 +0000 UTC m=+3288.161714487" watchObservedRunningTime="2026-11-03 21:24:08.160192831 +0000 UTC m=+3288.164151556"
Nov 03 21:24:09.059680 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:09.059641    5196 generic.go:356] "Generic (PLEG): container finished" podID="dafbb552-cee1-4efd-8209-a738b927ceec" containerID="3f9f6059c9ea4af6d9e84744db39871457700f5e38b2eab5b9ac040031427422" exitCode=0
Nov 03 21:24:09.060189 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:09.059682    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml" event={"ID":"dafbb552-cee1-4efd-8209-a738b927ceec","Type":"ContainerDied","Data":"3f9f6059c9ea4af6d9e84744db39871457700f5e38b2eab5b9ac040031427422"}
Nov 03 21:24:10.231452 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.231426    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:10.293598 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.293561    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-xks-ca\") pod \"dafbb552-cee1-4efd-8209-a738b927ceec\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") "
Nov 03 21:24:10.293812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.293664    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-openshift-ca\") pod \"dafbb552-cee1-4efd-8209-a738b927ceec\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") "
Nov 03 21:24:10.293812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.293704    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafbb552-cee1-4efd-8209-a738b927ceec-kube-api-access-849rw\" (UniqueName: \"kubernetes.io/projected/dafbb552-cee1-4efd-8209-a738b927ceec-kube-api-access-849rw\") pod \"dafbb552-cee1-4efd-8209-a738b927ceec\" (UID: \"dafbb552-cee1-4efd-8209-a738b927ceec\") "
Nov 03 21:24:10.294046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.294022    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-openshift-ca" pod "dafbb552-cee1-4efd-8209-a738b927ceec" (UID: "dafbb552-cee1-4efd-8209-a738b927ceec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:24:10.296110 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.296060    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-xks-ca" pod "dafbb552-cee1-4efd-8209-a738b927ceec" (UID: "dafbb552-cee1-4efd-8209-a738b927ceec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:24:10.296280 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.296258    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafbb552-cee1-4efd-8209-a738b927ceec-kube-api-access-849rw" pod "dafbb552-cee1-4efd-8209-a738b927ceec" (UID: "dafbb552-cee1-4efd-8209-a738b927ceec"). InnerVolumeSpecName "kube-api-access-849rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:24:10.395017 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.394971    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-849rw\" (UniqueName: \"kubernetes.io/projected/dafbb552-cee1-4efd-8209-a738b927ceec-kube-api-access-849rw\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:10.395017 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.395012    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:10.395291 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:10.395030    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafbb552-cee1-4efd-8209-a738b927ceec-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:11.073754 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:11.073721    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"
Nov 03 21:24:11.073955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:11.073755    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml" event={"ID":"dafbb552-cee1-4efd-8209-a738b927ceec","Type":"ContainerDied","Data":"8c48ade8ea251851e068a955026c18241c46050d0d99954278002f1486483ef1"}
Nov 03 21:24:11.073955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:11.073808    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c48ade8ea251851e068a955026c18241c46050d0d99954278002f1486483ef1"
Nov 03 21:24:18.484979 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:24:18.484938    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:24:20.747471 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:20.747436    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:24:20.754537 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:20.754500    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:24:20.765120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:20.765080    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:24:20.771412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:20.771390    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:24:28.355079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:28.355041    5196 scope.go:122] "RemoveContainer" containerID="050f42fb449502eba887a4e065bf8e60740540e401f2347b38c5ba4b6de5f7f5"
Nov 03 21:24:28.365053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:28.365030    5196 scope.go:122] "RemoveContainer" containerID="7c1e0f39bfcc4449c496c02e520c24816837b91024de9100bb34a7592287b791"
Nov 03 21:24:34.178797 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:34.178748    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:24:48.485033 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:24:48.484983    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:24:49.044857 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:49.044818    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"]
Nov 03 21:24:49.091500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:49.091462    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kmtjq"]
Nov 03 21:24:50.108409 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:50.108376    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472ad38a-7bd8-44e3-87ce-9ef3a84ee342" path="/var/lib/kubelet/pods/472ad38a-7bd8-44e3-87ce-9ef3a84ee342/volumes"
Nov 03 21:24:54.527471 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:54.527382    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:24:56.303527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.303490    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"]
Nov 03 21:24:56.304029 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.304015    5196 cpu_manager.go:414] "containerMap: removing container" podUID="dafbb552-cee1-4efd-8209-a738b927ceec" containerName="revoke-keys"
Nov 03 21:24:56.304076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.304030    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafbb552-cee1-4efd-8209-a738b927ceec" containerName="revoke-keys"
Nov 03 21:24:56.304167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.304121    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafbb552-cee1-4efd-8209-a738b927ceec" containerName="revoke-keys"
Nov 03 21:24:56.307630 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.307611    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.361319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.361260    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"]
Nov 03 21:24:56.404635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.404599    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:24:56.404841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.404767    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:24:56.418318 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.418285    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.418422 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.418403    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrbfd\" (UniqueName: \"kubernetes.io/projected/9aeb186f-38f8-4ae9-babd-849f58784875-kube-api-access-zrbfd\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.418471 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.418461    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.519585 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.519548    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrbfd\" (UniqueName: \"kubernetes.io/projected/9aeb186f-38f8-4ae9-babd-849f58784875-kube-api-access-zrbfd\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.519785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.519602    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.519785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.519674    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.520447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.520423    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.522225 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.522201    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.573669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.573628    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrbfd\" (UniqueName: \"kubernetes.io/projected/9aeb186f-38f8-4ae9-babd-849f58784875-kube-api-access-zrbfd\") pod \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.621319 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.621277    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:56.996715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:56.996625    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"]
Nov 03 21:24:57.003552 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:24:57.003516    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aeb186f_38f8_4ae9_babd_849f58784875.slice/crio-c7ae70eb90fe8baf8a77b0856c9eab74255f5ad7d2eb79259d7407653b373dbd WatchSource:0}: Error finding container c7ae70eb90fe8baf8a77b0856c9eab74255f5ad7d2eb79259d7407653b373dbd: Status 404 returned error can't find the container with id c7ae70eb90fe8baf8a77b0856c9eab74255f5ad7d2eb79259d7407653b373dbd
Nov 03 21:24:57.401355 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:57.401316    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2" event={"ID":"9aeb186f-38f8-4ae9-babd-849f58784875","Type":"ContainerStarted","Data":"fd904ce354892adc78690ab3c9e9d5b846d63c58e61a87f0413f9c5960f24c37"}
Nov 03 21:24:57.401755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:57.401374    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2" event={"ID":"9aeb186f-38f8-4ae9-babd-849f58784875","Type":"ContainerStarted","Data":"c7ae70eb90fe8baf8a77b0856c9eab74255f5ad7d2eb79259d7407653b373dbd"}
Nov 03 21:24:57.510686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:57.510625    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2" podStartSLOduration=1.5106097950000001 podStartE2EDuration="1.510609795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:57.507352831 +0000 UTC m=+3337.511311565" watchObservedRunningTime="2026-11-03 21:24:57.510609795 +0000 UTC m=+3337.514568550"
Nov 03 21:24:58.409784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:58.409743    5196 generic.go:356] "Generic (PLEG): container finished" podID="9aeb186f-38f8-4ae9-babd-849f58784875" containerID="fd904ce354892adc78690ab3c9e9d5b846d63c58e61a87f0413f9c5960f24c37" exitCode=0
Nov 03 21:24:58.410236 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:58.409829    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2" event={"ID":"9aeb186f-38f8-4ae9-babd-849f58784875","Type":"ContainerDied","Data":"fd904ce354892adc78690ab3c9e9d5b846d63c58e61a87f0413f9c5960f24c37"}
Nov 03 21:24:59.575276 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.575250    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:24:59.649247 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.649209    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aeb186f-38f8-4ae9-babd-849f58784875-kube-api-access-zrbfd\" (UniqueName: \"kubernetes.io/projected/9aeb186f-38f8-4ae9-babd-849f58784875-kube-api-access-zrbfd\") pod \"9aeb186f-38f8-4ae9-babd-849f58784875\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") "
Nov 03 21:24:59.649441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.649288    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-openshift-ca\") pod \"9aeb186f-38f8-4ae9-babd-849f58784875\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") "
Nov 03 21:24:59.649500 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.649456    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-xks-ca\") pod \"9aeb186f-38f8-4ae9-babd-849f58784875\" (UID: \"9aeb186f-38f8-4ae9-babd-849f58784875\") "
Nov 03 21:24:59.649656 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.649625    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-openshift-ca" pod "9aeb186f-38f8-4ae9-babd-849f58784875" (UID: "9aeb186f-38f8-4ae9-babd-849f58784875"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:24:59.649881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.649865    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:59.651636 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.651612    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-xks-ca" pod "9aeb186f-38f8-4ae9-babd-849f58784875" (UID: "9aeb186f-38f8-4ae9-babd-849f58784875"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:24:59.651709 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.651625    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aeb186f-38f8-4ae9-babd-849f58784875-kube-api-access-zrbfd" pod "9aeb186f-38f8-4ae9-babd-849f58784875" (UID: "9aeb186f-38f8-4ae9-babd-849f58784875"). InnerVolumeSpecName "kube-api-access-zrbfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:24:59.751352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.751264    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrbfd\" (UniqueName: \"kubernetes.io/projected/9aeb186f-38f8-4ae9-babd-849f58784875-kube-api-access-zrbfd\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:59.751352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:24:59.751299    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aeb186f-38f8-4ae9-babd-849f58784875-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:00.425734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:00.425643    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:25:00.425734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:00.425684    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2" event={"ID":"9aeb186f-38f8-4ae9-babd-849f58784875","Type":"ContainerDied","Data":"c7ae70eb90fe8baf8a77b0856c9eab74255f5ad7d2eb79259d7407653b373dbd"}
Nov 03 21:25:00.425734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:00.425734    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ae70eb90fe8baf8a77b0856c9eab74255f5ad7d2eb79259d7407653b373dbd"
Nov 03 21:25:07.003285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.003247    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"]
Nov 03 21:25:07.003793 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.003778    5196 cpu_manager.go:414] "containerMap: removing container" podUID="9aeb186f-38f8-4ae9-babd-849f58784875" containerName="revoke-keys"
Nov 03 21:25:07.003793 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.003792    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aeb186f-38f8-4ae9-babd-849f58784875" containerName="revoke-keys"
Nov 03 21:25:07.003893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.003883    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aeb186f-38f8-4ae9-babd-849f58784875" containerName="revoke-keys"
Nov 03 21:25:07.007502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.007480    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.061783 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.061730    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"]
Nov 03 21:25:07.102961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.102933    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:25:07.103152 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.102996    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:25:07.128086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.128052    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.128329 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.128251    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.128423 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.128401    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcnl\" (UniqueName: \"kubernetes.io/projected/d720c369-96e9-4320-99d5-e7ac462aef76-kube-api-access-7lcnl\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.229397 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.229362    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.229623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.229485    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcnl\" (UniqueName: \"kubernetes.io/projected/d720c369-96e9-4320-99d5-e7ac462aef76-kube-api-access-7lcnl\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.229623 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.229528    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.230238 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.230209    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.231930 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.231908    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.283557 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.283482    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcnl\" (UniqueName: \"kubernetes.io/projected/d720c369-96e9-4320-99d5-e7ac462aef76-kube-api-access-7lcnl\") pod \"maas-api-revoke-sub-5a29efb3-bvmm6\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.322503 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.322469    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:07.700969 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:07.700935    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"]
Nov 03 21:25:07.707440 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:25:07.707412    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd720c369_96e9_4320_99d5_e7ac462aef76.slice/crio-489483bb0930bb0e4edf8269c4f75115e0a555f21f686de6a1e277a74ca4c7be WatchSource:0}: Error finding container 489483bb0930bb0e4edf8269c4f75115e0a555f21f686de6a1e277a74ca4c7be: Status 404 returned error can't find the container with id 489483bb0930bb0e4edf8269c4f75115e0a555f21f686de6a1e277a74ca4c7be
Nov 03 21:25:08.480055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:08.480019    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6" event={"ID":"d720c369-96e9-4320-99d5-e7ac462aef76","Type":"ContainerStarted","Data":"ef521f7ffd8b57925ae0244dfb1778bda96ad7811cae9e478aad26363015d24e"}
Nov 03 21:25:08.480544 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:08.480068    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6" event={"ID":"d720c369-96e9-4320-99d5-e7ac462aef76","Type":"ContainerStarted","Data":"489483bb0930bb0e4edf8269c4f75115e0a555f21f686de6a1e277a74ca4c7be"}
Nov 03 21:25:08.590860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:08.590799    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6" podStartSLOduration=2.5907839729999997 podStartE2EDuration="2.590783973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:08.586957671 +0000 UTC m=+3348.590916396" watchObservedRunningTime="2026-11-03 21:25:08.590783973 +0000 UTC m=+3348.594742729"
Nov 03 21:25:09.058714 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:09.058678    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:25:09.491870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:09.491821    5196 generic.go:356] "Generic (PLEG): container finished" podID="d720c369-96e9-4320-99d5-e7ac462aef76" containerID="ef521f7ffd8b57925ae0244dfb1778bda96ad7811cae9e478aad26363015d24e" exitCode=0
Nov 03 21:25:09.492535 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:09.491908    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6" event={"ID":"d720c369-96e9-4320-99d5-e7ac462aef76","Type":"ContainerDied","Data":"ef521f7ffd8b57925ae0244dfb1778bda96ad7811cae9e478aad26363015d24e"}
Nov 03 21:25:10.917788 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:10.917749    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:10.922875 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:10.922850    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-openshift-ca\") pod \"d720c369-96e9-4320-99d5-e7ac462aef76\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") "
Nov 03 21:25:10.923022 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:10.922886    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-xks-ca\") pod \"d720c369-96e9-4320-99d5-e7ac462aef76\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") "
Nov 03 21:25:10.923022 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:10.922910    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d720c369-96e9-4320-99d5-e7ac462aef76-kube-api-access-7lcnl\" (UniqueName: \"kubernetes.io/projected/d720c369-96e9-4320-99d5-e7ac462aef76-kube-api-access-7lcnl\") pod \"d720c369-96e9-4320-99d5-e7ac462aef76\" (UID: \"d720c369-96e9-4320-99d5-e7ac462aef76\") "
Nov 03 21:25:10.923604 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:10.923571    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-openshift-ca" pod "d720c369-96e9-4320-99d5-e7ac462aef76" (UID: "d720c369-96e9-4320-99d5-e7ac462aef76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:25:10.925412 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:10.925384    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d720c369-96e9-4320-99d5-e7ac462aef76-kube-api-access-7lcnl" pod "d720c369-96e9-4320-99d5-e7ac462aef76" (UID: "d720c369-96e9-4320-99d5-e7ac462aef76"). InnerVolumeSpecName "kube-api-access-7lcnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:25:10.925507 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:10.925446    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-xks-ca" pod "d720c369-96e9-4320-99d5-e7ac462aef76" (UID: "d720c369-96e9-4320-99d5-e7ac462aef76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:25:11.023832 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:11.023790    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:11.023832 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:11.023825    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d720c369-96e9-4320-99d5-e7ac462aef76-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:11.023832 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:11.023841    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lcnl\" (UniqueName: \"kubernetes.io/projected/d720c369-96e9-4320-99d5-e7ac462aef76-kube-api-access-7lcnl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:11.518261 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:11.518233    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"
Nov 03 21:25:11.518261 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:11.518241    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6" event={"ID":"d720c369-96e9-4320-99d5-e7ac462aef76","Type":"ContainerDied","Data":"489483bb0930bb0e4edf8269c4f75115e0a555f21f686de6a1e277a74ca4c7be"}
Nov 03 21:25:11.518486 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:11.518284    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489483bb0930bb0e4edf8269c4f75115e0a555f21f686de6a1e277a74ca4c7be"
Nov 03 21:25:18.484690 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:25:18.484650    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:25:19.567348 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:19.567284    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:25:23.580761 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.580725    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"]
Nov 03 21:25:23.628730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.628694    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r9mml"]
Nov 03 21:25:23.711567 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.711535    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"]
Nov 03 21:25:23.712087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.712075    5196 cpu_manager.go:414] "containerMap: removing container" podUID="d720c369-96e9-4320-99d5-e7ac462aef76" containerName="revoke-keys"
Nov 03 21:25:23.712156 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.712109    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d720c369-96e9-4320-99d5-e7ac462aef76" containerName="revoke-keys"
Nov 03 21:25:23.712208 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.712198    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d720c369-96e9-4320-99d5-e7ac462aef76" containerName="revoke-keys"
Nov 03 21:25:23.715928 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.715908    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.747614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.747578    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.747785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.747626    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrg6\" (UniqueName: \"kubernetes.io/projected/f93eef37-cadf-4170-9840-ea9cb04aa7cb-kube-api-access-wdrg6\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.747785 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.747729    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.765503 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.765475    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:25:23.765730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.765605    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:25:23.768285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.768261    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"]
Nov 03 21:25:23.848487 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.848393    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.848487 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.848444    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrg6\" (UniqueName: \"kubernetes.io/projected/f93eef37-cadf-4170-9840-ea9cb04aa7cb-kube-api-access-wdrg6\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.848744 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.848498    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.849281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.849246    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.851015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.850991    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:23.902847 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:23.902818    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrg6\" (UniqueName: \"kubernetes.io/projected/f93eef37-cadf-4170-9840-ea9cb04aa7cb-kube-api-access-wdrg6\") pod \"maas-api-revoke-sub-3d4156f1-lk9k7\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:24.030133 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:24.030087    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:24.046262 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:24.046232    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"]
Nov 03 21:25:24.050893 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:24.050862    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5dwc8"]
Nov 03 21:25:24.115666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:24.115579    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b092c1-efc2-4735-8cfe-b81b0182d868" path="/var/lib/kubelet/pods/09b092c1-efc2-4735-8cfe-b81b0182d868/volumes"
Nov 03 21:25:24.116186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:24.116162    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafbb552-cee1-4efd-8209-a738b927ceec" path="/var/lib/kubelet/pods/dafbb552-cee1-4efd-8209-a738b927ceec/volumes"
Nov 03 21:25:24.410334 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:24.410244    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"]
Nov 03 21:25:24.417220 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:25:24.417189    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93eef37_cadf_4170_9840_ea9cb04aa7cb.slice/crio-e366d635f78f48d105af965741c6adb7da30087e6f609b35ff1fe0fe66a9fd17 WatchSource:0}: Error finding container e366d635f78f48d105af965741c6adb7da30087e6f609b35ff1fe0fe66a9fd17: Status 404 returned error can't find the container with id e366d635f78f48d105af965741c6adb7da30087e6f609b35ff1fe0fe66a9fd17
Nov 03 21:25:24.639991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:24.639961    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7" event={"ID":"f93eef37-cadf-4170-9840-ea9cb04aa7cb","Type":"ContainerStarted","Data":"e366d635f78f48d105af965741c6adb7da30087e6f609b35ff1fe0fe66a9fd17"}
Nov 03 21:25:25.648615 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:25.648579    5196 generic.go:356] "Generic (PLEG): container finished" podID="f93eef37-cadf-4170-9840-ea9cb04aa7cb" containerID="077bca1a17e9eea0402a01d77cd5de4fadc46feca11de10db1a2778de1c84f8b" exitCode=0
Nov 03 21:25:25.649029 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:25.648652    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7" event={"ID":"f93eef37-cadf-4170-9840-ea9cb04aa7cb","Type":"ContainerDied","Data":"077bca1a17e9eea0402a01d77cd5de4fadc46feca11de10db1a2778de1c84f8b"}
Nov 03 21:25:26.808863 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.808836    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:26.879397 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.879360    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-xks-ca\") pod \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") "
Nov 03 21:25:26.879596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.879415    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-openshift-ca\") pod \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") "
Nov 03 21:25:26.879596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.879471    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93eef37-cadf-4170-9840-ea9cb04aa7cb-kube-api-access-wdrg6\" (UniqueName: \"kubernetes.io/projected/f93eef37-cadf-4170-9840-ea9cb04aa7cb-kube-api-access-wdrg6\") pod \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\" (UID: \"f93eef37-cadf-4170-9840-ea9cb04aa7cb\") "
Nov 03 21:25:26.879821 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.879794    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-openshift-ca" pod "f93eef37-cadf-4170-9840-ea9cb04aa7cb" (UID: "f93eef37-cadf-4170-9840-ea9cb04aa7cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:25:26.879941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.879926    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:26.881837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.881807    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-xks-ca" pod "f93eef37-cadf-4170-9840-ea9cb04aa7cb" (UID: "f93eef37-cadf-4170-9840-ea9cb04aa7cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:25:26.881955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.881937    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93eef37-cadf-4170-9840-ea9cb04aa7cb-kube-api-access-wdrg6" pod "f93eef37-cadf-4170-9840-ea9cb04aa7cb" (UID: "f93eef37-cadf-4170-9840-ea9cb04aa7cb"). InnerVolumeSpecName "kube-api-access-wdrg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:25:26.981396 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.981294    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93eef37-cadf-4170-9840-ea9cb04aa7cb-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:26.981396 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:26.981330    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdrg6\" (UniqueName: \"kubernetes.io/projected/f93eef37-cadf-4170-9840-ea9cb04aa7cb-kube-api-access-wdrg6\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:27.669034 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:27.669001    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"
Nov 03 21:25:27.669261 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:27.669033    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7" event={"ID":"f93eef37-cadf-4170-9840-ea9cb04aa7cb","Type":"ContainerDied","Data":"e366d635f78f48d105af965741c6adb7da30087e6f609b35ff1fe0fe66a9fd17"}
Nov 03 21:25:27.669261 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:27.669111    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e366d635f78f48d105af965741c6adb7da30087e6f609b35ff1fe0fe66a9fd17"
Nov 03 21:25:39.683180 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:39.683134    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:41.594538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:41.594488    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:44.329202 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:44.329165    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:44.484677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:44.484637    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:44.639837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:44.639739    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:46.046846 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:46.046807    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"]
Nov 03 21:25:46.093379 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:46.093335    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-m8wk5"]
Nov 03 21:25:46.109359 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:46.109314    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ff8a31-394e-432a-b498-ca3933d657ee" path="/var/lib/kubelet/pods/21ff8a31-394e-432a-b498-ca3933d657ee/volumes"
Nov 03 21:25:47.516617 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:47.516585    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:47.687736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:47.687699    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:48.485322 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:25:48.485284    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:25:50.304581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:50.304541    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:50.452406 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:50.452368    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:50.597508 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:50.597469    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:53.202749 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:53.202710    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:54.774735 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:54.774702    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:54.965759 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:54.965512    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:57.892853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:57.892808    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:25:59.438887 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:25:59.438844    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:03.935550 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:03.935511    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:08.722366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:08.722319    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:09.122978 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:09.122940    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:09.687609 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:09.687563    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:09.893109 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:09.893056    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:12.589932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:12.589894    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:13.320005 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:13.319962    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:17.067765 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:17.067722    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:26:17.859788 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:17.859750    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:18.485026 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:26:18.484989    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:26:22.363773 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:22.363698    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:26.826083 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:26.826040    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:28.407107 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:28.407047    5196 scope.go:122] "RemoveContainer" containerID="e5f5a6a60e6725a0afb8a2912bfcbac330f2f0e639fe72f7d295def1823404be"
Nov 03 21:26:28.417512 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:28.417487    5196 scope.go:122] "RemoveContainer" containerID="5d0ff92b70e1880d4d2991b50a7dd220962d08f29e86342ba4bec01cbce32597"
Nov 03 21:26:31.243045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:31.243007    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:33.042848 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:33.042815    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"]
Nov 03 21:26:33.089683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:33.089640    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fcwbr"]
Nov 03 21:26:34.108484 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:34.108452    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036e543d-0cbf-4a4c-b978-fe8dbaca410b" path="/var/lib/kubelet/pods/036e543d-0cbf-4a4c-b978-fe8dbaca410b/volumes"
Nov 03 21:26:35.826231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:35.826187    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:40.236280 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:40.236228    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:42.658314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:42.658273    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:26:44.047727 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:44.047689    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"]
Nov 03 21:26:44.096800 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:44.096755    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vftxl"]
Nov 03 21:26:44.110216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:44.110186    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc73368c-c20c-4b05-bf52-3020a1bac6ae" path="/var/lib/kubelet/pods/dc73368c-c20c-4b05-bf52-3020a1bac6ae/volumes"
Nov 03 21:26:44.747366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:44.747334    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:48.486501 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:26:48.486465    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:26:49.240432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:49.240394    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:53.728225 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:53.728185    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:55.117389 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:55.117355    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:55.262296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:55.262260    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:56.962107 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:56.962060    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"]
Nov 03 21:26:56.962630 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:56.962615    5196 cpu_manager.go:414] "containerMap: removing container" podUID="f93eef37-cadf-4170-9840-ea9cb04aa7cb" containerName="revoke-keys"
Nov 03 21:26:56.962677 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:56.962631    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93eef37-cadf-4170-9840-ea9cb04aa7cb" containerName="revoke-keys"
Nov 03 21:26:56.962721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:56.962714    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93eef37-cadf-4170-9840-ea9cb04aa7cb" containerName="revoke-keys"
Nov 03 21:26:56.966066 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:56.966049    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.016357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.016173    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:57.016357 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.016321    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:26:57.018929 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.018902    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"]
Nov 03 21:26:57.029053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.029024    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.029229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.029072    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.029229 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.029170    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjj5\" (UniqueName: \"kubernetes.io/projected/6da152cc-5f4e-40c3-80df-d9237dd1eda0-kube-api-access-qjjj5\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.130709 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.130676    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.130905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.130719    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.130905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.130745    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjj5\" (UniqueName: \"kubernetes.io/projected/6da152cc-5f4e-40c3-80df-d9237dd1eda0-kube-api-access-qjjj5\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.131399 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.131374    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.133448 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.133424    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.184985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.184956    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjj5\" (UniqueName: \"kubernetes.io/projected/6da152cc-5f4e-40c3-80df-d9237dd1eda0-kube-api-access-qjjj5\") pod \"maas-api-revoke-sub-40786756-8rvlc\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.279235 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.279149    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:26:57.655250 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:57.655213    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"]
Nov 03 21:26:57.661966 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:26:57.661939    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da152cc_5f4e_40c3_80df_d9237dd1eda0.slice/crio-08684103c8ba035991059965066142f9c28b5d09fc1449130a8f3b5fe4c62b87 WatchSource:0}: Error finding container 08684103c8ba035991059965066142f9c28b5d09fc1449130a8f3b5fe4c62b87: Status 404 returned error can't find the container with id 08684103c8ba035991059965066142f9c28b5d09fc1449130a8f3b5fe4c62b87
Nov 03 21:26:58.223009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:58.222910    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:58.287866 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:58.287833    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc" event={"ID":"6da152cc-5f4e-40c3-80df-d9237dd1eda0","Type":"ContainerStarted","Data":"9d128d603283778c5f8bf88e8fc83f7c216a30de1790ab957638281e6154bd8f"}
Nov 03 21:26:58.288066 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:58.287886    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc" event={"ID":"6da152cc-5f4e-40c3-80df-d9237dd1eda0","Type":"ContainerStarted","Data":"08684103c8ba035991059965066142f9c28b5d09fc1449130a8f3b5fe4c62b87"}
Nov 03 21:26:58.394805 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:58.394752    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc" podStartSLOduration=2.394738708 podStartE2EDuration="2.394738708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:58.394302372 +0000 UTC m=+3458.398261107" watchObservedRunningTime="2026-11-03 21:26:58.394738708 +0000 UTC m=+3458.398697431"
Nov 03 21:26:59.297737 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:59.297696    5196 generic.go:356] "Generic (PLEG): container finished" podID="6da152cc-5f4e-40c3-80df-d9237dd1eda0" containerID="9d128d603283778c5f8bf88e8fc83f7c216a30de1790ab957638281e6154bd8f" exitCode=0
Nov 03 21:26:59.298262 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:26:59.297777    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc" event={"ID":"6da152cc-5f4e-40c3-80df-d9237dd1eda0","Type":"ContainerDied","Data":"9d128d603283778c5f8bf88e8fc83f7c216a30de1790ab957638281e6154bd8f"}
Nov 03 21:27:00.458124 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.458062    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:27:00.567160 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.567050    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-openshift-ca\") pod \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") "
Nov 03 21:27:00.567343 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.567159    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-xks-ca\") pod \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") "
Nov 03 21:27:00.567567 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.567541    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-openshift-ca" pod "6da152cc-5f4e-40c3-80df-d9237dd1eda0" (UID: "6da152cc-5f4e-40c3-80df-d9237dd1eda0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:00.567705 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.567681    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da152cc-5f4e-40c3-80df-d9237dd1eda0-kube-api-access-qjjj5\" (UniqueName: \"kubernetes.io/projected/6da152cc-5f4e-40c3-80df-d9237dd1eda0-kube-api-access-qjjj5\") pod \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\" (UID: \"6da152cc-5f4e-40c3-80df-d9237dd1eda0\") "
Nov 03 21:27:00.568377 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.568351    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:00.569921 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.569895    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-xks-ca" pod "6da152cc-5f4e-40c3-80df-d9237dd1eda0" (UID: "6da152cc-5f4e-40c3-80df-d9237dd1eda0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:00.570628 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.570594    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da152cc-5f4e-40c3-80df-d9237dd1eda0-kube-api-access-qjjj5" pod "6da152cc-5f4e-40c3-80df-d9237dd1eda0" (UID: "6da152cc-5f4e-40c3-80df-d9237dd1eda0"). InnerVolumeSpecName "kube-api-access-qjjj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:00.669594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.669551    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da152cc-5f4e-40c3-80df-d9237dd1eda0-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:00.669594 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:00.669588    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjj5\" (UniqueName: \"kubernetes.io/projected/6da152cc-5f4e-40c3-80df-d9237dd1eda0-kube-api-access-qjjj5\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:01.312752 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:01.312719    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"
Nov 03 21:27:01.312954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:01.312753    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc" event={"ID":"6da152cc-5f4e-40c3-80df-d9237dd1eda0","Type":"ContainerDied","Data":"08684103c8ba035991059965066142f9c28b5d09fc1449130a8f3b5fe4c62b87"}
Nov 03 21:27:01.312954 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:01.312803    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08684103c8ba035991059965066142f9c28b5d09fc1449130a8f3b5fe4c62b87"
Nov 03 21:27:02.764782 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:02.764744    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:07.226591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:07.226551    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:08.573344 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:08.573311    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:08.986010 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:08.985928    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"]
Nov 03 21:27:08.986499 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:08.986484    5196 cpu_manager.go:414] "containerMap: removing container" podUID="6da152cc-5f4e-40c3-80df-d9237dd1eda0" containerName="revoke-keys"
Nov 03 21:27:08.986545 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:08.986500    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da152cc-5f4e-40c3-80df-d9237dd1eda0" containerName="revoke-keys"
Nov 03 21:27:08.986587 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:08.986572    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da152cc-5f4e-40c3-80df-d9237dd1eda0" containerName="revoke-keys"
Nov 03 21:27:08.990044 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:08.990022    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.043519 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.043488    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"]
Nov 03 21:27:09.062779 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.062734    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jldl\" (UniqueName: \"kubernetes.io/projected/c2bccc98-2c24-4d5a-b791-1786ebb41082-kube-api-access-7jldl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.062968 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.062795    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.063104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.063057    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.086663 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.086629    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:27:09.086663 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.086647    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:27:09.164192 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.164147    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jldl\" (UniqueName: \"kubernetes.io/projected/c2bccc98-2c24-4d5a-b791-1786ebb41082-kube-api-access-7jldl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.164396 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.164215    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.164396 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.164330    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.165059 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.165024    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.166867 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.166842    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.218315 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.218273    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jldl\" (UniqueName: \"kubernetes.io/projected/c2bccc98-2c24-4d5a-b791-1786ebb41082-kube-api-access-7jldl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.303688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.303603    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:09.677432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:09.677395    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"]
Nov 03 21:27:09.684188 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:27:09.684154    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bccc98_2c24_4d5a_b791_1786ebb41082.slice/crio-2a9ac2a823ae4df1290a44f43ed456244d14a03d1c6e031d4eb4c8f93aeac3aa WatchSource:0}: Error finding container 2a9ac2a823ae4df1290a44f43ed456244d14a03d1c6e031d4eb4c8f93aeac3aa: Status 404 returned error can't find the container with id 2a9ac2a823ae4df1290a44f43ed456244d14a03d1c6e031d4eb4c8f93aeac3aa
Nov 03 21:27:10.379980 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:10.379939    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9" event={"ID":"c2bccc98-2c24-4d5a-b791-1786ebb41082","Type":"ContainerStarted","Data":"8ca43bedd57b53492ec43228a476679287126333a774878a37b2a20b4a2e7859"}
Nov 03 21:27:10.380190 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:10.379997    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9" event={"ID":"c2bccc98-2c24-4d5a-b791-1786ebb41082","Type":"ContainerStarted","Data":"2a9ac2a823ae4df1290a44f43ed456244d14a03d1c6e031d4eb4c8f93aeac3aa"}
Nov 03 21:27:10.489408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:10.489353    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9" podStartSLOduration=2.489338171 podStartE2EDuration="2.489338171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:10.486655829 +0000 UTC m=+3470.490614574" watchObservedRunningTime="2026-11-03 21:27:10.489338171 +0000 UTC m=+3470.493296894"
Nov 03 21:27:11.389459 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:11.389424    5196 generic.go:356] "Generic (PLEG): container finished" podID="c2bccc98-2c24-4d5a-b791-1786ebb41082" containerID="8ca43bedd57b53492ec43228a476679287126333a774878a37b2a20b4a2e7859" exitCode=0
Nov 03 21:27:11.389995 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:11.389513    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9" event={"ID":"c2bccc98-2c24-4d5a-b791-1786ebb41082","Type":"ContainerDied","Data":"8ca43bedd57b53492ec43228a476679287126333a774878a37b2a20b4a2e7859"}
Nov 03 21:27:11.565285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:11.565241    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:27:12.030024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.029989    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:12.546802 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.546774    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:12.708716 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.708612    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bccc98-2c24-4d5a-b791-1786ebb41082-kube-api-access-7jldl\" (UniqueName: \"kubernetes.io/projected/c2bccc98-2c24-4d5a-b791-1786ebb41082-kube-api-access-7jldl\") pod \"c2bccc98-2c24-4d5a-b791-1786ebb41082\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") "
Nov 03 21:27:12.708716 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.708692    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-xks-ca\") pod \"c2bccc98-2c24-4d5a-b791-1786ebb41082\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") "
Nov 03 21:27:12.708716 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.708712    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-openshift-ca\") pod \"c2bccc98-2c24-4d5a-b791-1786ebb41082\" (UID: \"c2bccc98-2c24-4d5a-b791-1786ebb41082\") "
Nov 03 21:27:12.709288 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.709249    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-openshift-ca" pod "c2bccc98-2c24-4d5a-b791-1786ebb41082" (UID: "c2bccc98-2c24-4d5a-b791-1786ebb41082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:12.711159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.711126    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-xks-ca" pod "c2bccc98-2c24-4d5a-b791-1786ebb41082" (UID: "c2bccc98-2c24-4d5a-b791-1786ebb41082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:12.711256 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.711160    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bccc98-2c24-4d5a-b791-1786ebb41082-kube-api-access-7jldl" pod "c2bccc98-2c24-4d5a-b791-1786ebb41082" (UID: "c2bccc98-2c24-4d5a-b791-1786ebb41082"). InnerVolumeSpecName "kube-api-access-7jldl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:12.810312 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.810276    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:12.810312 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.810311    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2bccc98-2c24-4d5a-b791-1786ebb41082-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:12.810510 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:12.810327    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jldl\" (UniqueName: \"kubernetes.io/projected/c2bccc98-2c24-4d5a-b791-1786ebb41082-kube-api-access-7jldl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:13.405426 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:13.405396    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"
Nov 03 21:27:13.405426 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:13.405415    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9" event={"ID":"c2bccc98-2c24-4d5a-b791-1786ebb41082","Type":"ContainerDied","Data":"2a9ac2a823ae4df1290a44f43ed456244d14a03d1c6e031d4eb4c8f93aeac3aa"}
Nov 03 21:27:13.405724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:13.405471    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a9ac2a823ae4df1290a44f43ed456244d14a03d1c6e031d4eb4c8f93aeac3aa"
Nov 03 21:27:13.568673 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:13.568620    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:16.737371 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:16.737323    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:18.485220 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:27:18.485182    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:27:20.046755 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:20.046715    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"]
Nov 03 21:27:20.093519 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:20.093478    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vtngv"]
Nov 03 21:27:20.108193 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:20.108164    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a51a5500-420c-49b0-a25e-e81c369ca617" path="/var/lib/kubelet/pods/a51a5500-420c-49b0-a25e-e81c369ca617/volumes"
Nov 03 21:27:20.678182 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:20.678144    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:20.877211 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:20.877172    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:21.031079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:21.030994    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:24.836447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:24.836399    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:25.095960 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:25.095868    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:25.382609 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:25.382513    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"]
Nov 03 21:27:25.429354 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:25.429309    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9f39a-rplx9"]
Nov 03 21:27:25.846709 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:25.846670    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:26.108845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:26.108760    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bccc98-2c24-4d5a-b791-1786ebb41082" path="/var/lib/kubelet/pods/c2bccc98-2c24-4d5a-b791-1786ebb41082/volumes"
Nov 03 21:27:27.705356 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:27.705318    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:27:28.447762 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:28.447730    5196 scope.go:122] "RemoveContainer" containerID="be3d9c4afbca5d6ee04cbb5210deacf95305605a35da16aa18fd13cec604b880"
Nov 03 21:27:31.444571 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:31.444527    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:31.598503 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:31.598470    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:31.756813 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:31.756723    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:43.189907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.189866    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"]
Nov 03 21:27:43.235436 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.235392    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lk9k7"]
Nov 03 21:27:43.293063 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.293031    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"]
Nov 03 21:27:43.293615 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.293599    5196 cpu_manager.go:414] "containerMap: removing container" podUID="c2bccc98-2c24-4d5a-b791-1786ebb41082" containerName="revoke-keys"
Nov 03 21:27:43.293685 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.293617    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bccc98-2c24-4d5a-b791-1786ebb41082" containerName="revoke-keys"
Nov 03 21:27:43.293731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.293723    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bccc98-2c24-4d5a-b791-1786ebb41082" containerName="revoke-keys"
Nov 03 21:27:43.297159 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.297141    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.347024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.346979    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.347314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.347165    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:27:43.347314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.347236    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:27:43.347314 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.347293    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.347506 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.347325    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgk2m\" (UniqueName: \"kubernetes.io/projected/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-kube-api-access-xgk2m\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.349411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.349389    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"]
Nov 03 21:27:43.448315 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.448206    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.448315 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.448267    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.448315 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.448292    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgk2m\" (UniqueName: \"kubernetes.io/projected/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-kube-api-access-xgk2m\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.448900 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.448865    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.451011 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.450984    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.502734 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.502698    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgk2m\" (UniqueName: \"kubernetes.io/projected/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-kube-api-access-xgk2m\") pod \"maas-api-revoke-sub-3d4156f1-76d9l\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.611501 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.611466    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:43.983430 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:43.983396    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"]
Nov 03 21:27:43.990196 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:27:43.990169    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae2065e_32f7_4c3d_9c13_cedd97e0d79e.slice/crio-908006e91d9fce915cc31d784b21879c2272f6b9abc8b4ce7ca8a72f99666f91 WatchSource:0}: Error finding container 908006e91d9fce915cc31d784b21879c2272f6b9abc8b4ce7ca8a72f99666f91: Status 404 returned error can't find the container with id 908006e91d9fce915cc31d784b21879c2272f6b9abc8b4ce7ca8a72f99666f91
Nov 03 21:27:44.109421 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:44.109384    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93eef37-cadf-4170-9840-ea9cb04aa7cb" path="/var/lib/kubelet/pods/f93eef37-cadf-4170-9840-ea9cb04aa7cb/volumes"
Nov 03 21:27:44.620371 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:44.620338    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l" event={"ID":"bae2065e-32f7-4c3d-9c13-cedd97e0d79e","Type":"ContainerStarted","Data":"4d56474102cf2d899bc76f132be541b5352b636b951e68071511fdd601489f74"}
Nov 03 21:27:44.620771 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:44.620386    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l" event={"ID":"bae2065e-32f7-4c3d-9c13-cedd97e0d79e","Type":"ContainerStarted","Data":"908006e91d9fce915cc31d784b21879c2272f6b9abc8b4ce7ca8a72f99666f91"}
Nov 03 21:27:44.727713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:44.727656    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l" podStartSLOduration=1.727638797 podStartE2EDuration="1.727638797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:44.726264756 +0000 UTC m=+3504.730223506" watchObservedRunningTime="2026-11-03 21:27:44.727638797 +0000 UTC m=+3504.731597537"
Nov 03 21:27:45.627858 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:45.627821    5196 generic.go:356] "Generic (PLEG): container finished" podID="bae2065e-32f7-4c3d-9c13-cedd97e0d79e" containerID="4d56474102cf2d899bc76f132be541b5352b636b951e68071511fdd601489f74" exitCode=0
Nov 03 21:27:45.628294 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:45.627922    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l" event={"ID":"bae2065e-32f7-4c3d-9c13-cedd97e0d79e","Type":"ContainerDied","Data":"4d56474102cf2d899bc76f132be541b5352b636b951e68071511fdd601489f74"}
Nov 03 21:27:46.787733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.787707    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:46.886176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.886075    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-openshift-ca\") pod \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") "
Nov 03 21:27:46.886176 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.886138    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-kube-api-access-xgk2m\" (UniqueName: \"kubernetes.io/projected/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-kube-api-access-xgk2m\") pod \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") "
Nov 03 21:27:46.886434 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.886286    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-xks-ca\") pod \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\" (UID: \"bae2065e-32f7-4c3d-9c13-cedd97e0d79e\") "
Nov 03 21:27:46.886545 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.886513    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-openshift-ca" pod "bae2065e-32f7-4c3d-9c13-cedd97e0d79e" (UID: "bae2065e-32f7-4c3d-9c13-cedd97e0d79e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:46.888526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.888494    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-kube-api-access-xgk2m" pod "bae2065e-32f7-4c3d-9c13-cedd97e0d79e" (UID: "bae2065e-32f7-4c3d-9c13-cedd97e0d79e"). InnerVolumeSpecName "kube-api-access-xgk2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:46.888660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.888608    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-xks-ca" pod "bae2065e-32f7-4c3d-9c13-cedd97e0d79e" (UID: "bae2065e-32f7-4c3d-9c13-cedd97e0d79e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:46.987902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.987857    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:46.987902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.987893    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:46.987902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:46.987904    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgk2m\" (UniqueName: \"kubernetes.io/projected/bae2065e-32f7-4c3d-9c13-cedd97e0d79e-kube-api-access-xgk2m\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:47.043671 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:47.043631    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"]
Nov 03 21:27:47.046644 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:47.046613    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hbsdk"]
Nov 03 21:27:47.642845 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:47.642816    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"
Nov 03 21:27:47.643015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:47.642852    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l" event={"ID":"bae2065e-32f7-4c3d-9c13-cedd97e0d79e","Type":"ContainerDied","Data":"908006e91d9fce915cc31d784b21879c2272f6b9abc8b4ce7ca8a72f99666f91"}
Nov 03 21:27:47.643015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:47.642895    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="908006e91d9fce915cc31d784b21879c2272f6b9abc8b4ce7ca8a72f99666f91"
Nov 03 21:27:48.109104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:48.109056    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceabb2eb-9b79-4fe4-82db-b0e8465ac623" path="/var/lib/kubelet/pods/ceabb2eb-9b79-4fe4-82db-b0e8465ac623/volumes"
Nov 03 21:27:48.485052 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:27:48.484952    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:27:50.368853 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:50.368816    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:50.518526 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:50.518494    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:55.427829 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:55.427727    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:56.109724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:27:56.109665    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:28:17.044033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:17.043997    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"]
Nov 03 21:28:17.090784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:17.090748    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-sqdvb"]
Nov 03 21:28:18.108956 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:18.108920    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a684792a-f808-4d8a-ba50-9e089dfe6194" path="/var/lib/kubelet/pods/a684792a-f808-4d8a-ba50-9e089dfe6194/volumes"
Nov 03 21:28:18.485518 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:28:18.485433    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:28:28.478671 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:28.478633    5196 scope.go:122] "RemoveContainer" containerID="6bebdf4cafb03ed485d2bfd6736e527af0afc76525a18f0483134f0ec884c1ab"
Nov 03 21:28:28.489088 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:28.489062    5196 scope.go:122] "RemoveContainer" containerID="5b0a2d02b97079dd371d1e0430184363a32b3e61c75f8e07c1bf88c0cc9cb361"
Nov 03 21:28:28.499258 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:28.499235    5196 scope.go:122] "RemoveContainer" containerID="05bd65ef8eea75f3eeae9b50eb7137d8b3dbfc9e27684170404c3b7250402230"
Nov 03 21:28:38.052400 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:38.052366    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"]
Nov 03 21:28:38.054966 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:38.054937    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-78vfh"]
Nov 03 21:28:38.108699 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:38.108664    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645a0b25-344d-4567-91ba-1d95b0a24f9b" path="/var/lib/kubelet/pods/645a0b25-344d-4567-91ba-1d95b0a24f9b/volumes"
Nov 03 21:28:48.485161 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:28:48.485121    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:28:55.768736 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:28:55.768684    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:29:05.495898 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.495861    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"]
Nov 03 21:29:05.496466 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.496451    5196 cpu_manager.go:414] "containerMap: removing container" podUID="bae2065e-32f7-4c3d-9c13-cedd97e0d79e" containerName="revoke-keys"
Nov 03 21:29:05.496514 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.496468    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae2065e-32f7-4c3d-9c13-cedd97e0d79e" containerName="revoke-keys"
Nov 03 21:29:05.496601 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.496591    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae2065e-32f7-4c3d-9c13-cedd97e0d79e" containerName="revoke-keys"
Nov 03 21:29:05.499984 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.499967    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:05.549046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.549011    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"]
Nov 03 21:29:05.593033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.592996    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jjnk7\""
Nov 03 21:29:05.627569 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.627530    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c78e9d1-87dc-417f-9b2e-87605826bb0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mc5f5\" (UID: \"6c78e9d1-87dc-417f-9b2e-87605826bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:05.627748 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.627684    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9xr\" (UniqueName: \"kubernetes.io/projected/6c78e9d1-87dc-417f-9b2e-87605826bb0a-kube-api-access-tj9xr\") pod \"kuadrant-operator-controller-manager-678d87766f-mc5f5\" (UID: \"6c78e9d1-87dc-417f-9b2e-87605826bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:05.728873 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.728836    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c78e9d1-87dc-417f-9b2e-87605826bb0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mc5f5\" (UID: \"6c78e9d1-87dc-417f-9b2e-87605826bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:05.729058 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.728938    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9xr\" (UniqueName: \"kubernetes.io/projected/6c78e9d1-87dc-417f-9b2e-87605826bb0a-kube-api-access-tj9xr\") pod \"kuadrant-operator-controller-manager-678d87766f-mc5f5\" (UID: \"6c78e9d1-87dc-417f-9b2e-87605826bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:05.729281 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.729252    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c78e9d1-87dc-417f-9b2e-87605826bb0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mc5f5\" (UID: \"6c78e9d1-87dc-417f-9b2e-87605826bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:05.783278 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.783206    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9xr\" (UniqueName: \"kubernetes.io/projected/6c78e9d1-87dc-417f-9b2e-87605826bb0a-kube-api-access-tj9xr\") pod \"kuadrant-operator-controller-manager-678d87766f-mc5f5\" (UID: \"6c78e9d1-87dc-417f-9b2e-87605826bb0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:05.815044 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:05.815013    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:06.311212 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:29:06.311183    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c78e9d1_87dc_417f_9b2e_87605826bb0a.slice/crio-f6ab1461dc6daf21b7a802a1cd625eee31b7877c2b1d84b4853786b02ba79140 WatchSource:0}: Error finding container f6ab1461dc6daf21b7a802a1cd625eee31b7877c2b1d84b4853786b02ba79140: Status 404 returned error can't find the container with id f6ab1461dc6daf21b7a802a1cd625eee31b7877c2b1d84b4853786b02ba79140
Nov 03 21:29:06.350940 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:06.350908    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"]
Nov 03 21:29:07.170640 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:07.170595    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5" event={"ID":"6c78e9d1-87dc-417f-9b2e-87605826bb0a","Type":"ContainerStarted","Data":"3e1d64bc81655c84787446a6832304ac23008dd59c7beb908475a8347a6dec4e"}
Nov 03 21:29:07.171036 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:07.170776    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:07.171036 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:07.170813    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5" event={"ID":"6c78e9d1-87dc-417f-9b2e-87605826bb0a","Type":"ContainerStarted","Data":"f6ab1461dc6daf21b7a802a1cd625eee31b7877c2b1d84b4853786b02ba79140"}
Nov 03 21:29:07.375489 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:07.375434    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5" podStartSLOduration=2.37541798 podStartE2EDuration="2.37541798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:07.327473641 +0000 UTC m=+3587.331432364" watchObservedRunningTime="2026-11-03 21:29:07.37541798 +0000 UTC m=+3587.379376704"
Nov 03 21:29:18.179435 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:18.179395    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mc5f5"
Nov 03 21:29:18.484531 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:29:18.484443    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:29:20.797581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:20.797554    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:29:20.804087 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:20.804063    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:29:20.816282 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:20.816254    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:29:20.822352 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:20.822329    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:29:28.430153 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:28.430106    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:29:28.535695 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:28.535659    5196 scope.go:122] "RemoveContainer" containerID="0bc96037240d37973ef46b4ade0bf1493e023d53857d3a0cd8ded15f3a3912e7"
Nov 03 21:29:28.546049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:28.546027    5196 scope.go:122] "RemoveContainer" containerID="072613f7bed19a9e243dd95019bdd69565dc364a8eb40d425cf273c6db54aca2"
Nov 03 21:29:45.661216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:45.661169    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:29:48.486933 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:29:48.486894    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:29:56.165132 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:56.165076    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:29:56.312884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:56.312841    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:29:58.159651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:29:58.159614    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:30:00.073121 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:00.064633    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"]
Nov 03 21:30:00.120867 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:00.120788    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"]
Nov 03 21:30:00.195672 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:30:00.195625    5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-e2fc9c5b-sxsg2\" is forbidden: User \"system:node:ip-10-0-128-49.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-49.us-west-2.compute.internal' and this object" podUID="9aeb186f-38f8-4ae9-babd-849f58784875" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sxsg2"
Nov 03 21:30:02.108874 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:02.108838    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aeb186f-38f8-4ae9-babd-849f58784875" path="/var/lib/kubelet/pods/9aeb186f-38f8-4ae9-babd-849f58784875/volumes"
Nov 03 21:30:05.356884 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.356847    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"]
Nov 03 21:30:05.361020 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.360998    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.413008 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.412968    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"]
Nov 03 21:30:05.456629 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.456601    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:30:05.456815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.456710    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:30:05.465407 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.465372    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.465566 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.465455    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4c2\" (UniqueName: \"kubernetes.io/projected/8f9ca5d2-617a-4e8a-ac80-743bf430e603-kube-api-access-4l4c2\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.465651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.465629    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.566291 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.566248    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.566493 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.566343    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.566493 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.566366    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4c2\" (UniqueName: \"kubernetes.io/projected/8f9ca5d2-617a-4e8a-ac80-743bf430e603-kube-api-access-4l4c2\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.567085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.566999    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.568846 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.568825    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.620213 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.620143    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4c2\" (UniqueName: \"kubernetes.io/projected/8f9ca5d2-617a-4e8a-ac80-743bf430e603-kube-api-access-4l4c2\") pod \"maas-api-revoke-sub-d2bc93bc-zzpp4\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:05.674312 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:05.674274    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:06.059613 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:06.059530    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"]
Nov 03 21:30:06.066381 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:30:06.066347    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9ca5d2_617a_4e8a_ac80_743bf430e603.slice/crio-7c6860bfcf12cfc28198fe671c87b6bd8e8b536737a212e5822b4e0e13b911f9 WatchSource:0}: Error finding container 7c6860bfcf12cfc28198fe671c87b6bd8e8b536737a212e5822b4e0e13b911f9: Status 404 returned error can't find the container with id 7c6860bfcf12cfc28198fe671c87b6bd8e8b536737a212e5822b4e0e13b911f9
Nov 03 21:30:06.616349 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:06.616247    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4" event={"ID":"8f9ca5d2-617a-4e8a-ac80-743bf430e603","Type":"ContainerStarted","Data":"8e198f302c2b4c1259658ba4addab33b504ab43cc17dfd5c1f0f5d2e599198e5"}
Nov 03 21:30:06.616349 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:06.616311    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4" event={"ID":"8f9ca5d2-617a-4e8a-ac80-743bf430e603","Type":"ContainerStarted","Data":"7c6860bfcf12cfc28198fe671c87b6bd8e8b536737a212e5822b4e0e13b911f9"}
Nov 03 21:30:06.726222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:06.726167    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4" podStartSLOduration=1.726151132 podStartE2EDuration="1.726151132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:06.723509812 +0000 UTC m=+3646.727468556" watchObservedRunningTime="2026-11-03 21:30:06.726151132 +0000 UTC m=+3646.730109855"
Nov 03 21:30:07.625694 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:07.625659    5196 generic.go:356] "Generic (PLEG): container finished" podID="8f9ca5d2-617a-4e8a-ac80-743bf430e603" containerID="8e198f302c2b4c1259658ba4addab33b504ab43cc17dfd5c1f0f5d2e599198e5" exitCode=0
Nov 03 21:30:07.626167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:07.625751    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4" event={"ID":"8f9ca5d2-617a-4e8a-ac80-743bf430e603","Type":"ContainerDied","Data":"8e198f302c2b4c1259658ba4addab33b504ab43cc17dfd5c1f0f5d2e599198e5"}
Nov 03 21:30:08.790715 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:08.790683    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:08.903056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:08.902958    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-openshift-ca\") pod \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") "
Nov 03 21:30:08.903056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:08.903011    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9ca5d2-617a-4e8a-ac80-743bf430e603-kube-api-access-4l4c2\" (UniqueName: \"kubernetes.io/projected/8f9ca5d2-617a-4e8a-ac80-743bf430e603-kube-api-access-4l4c2\") pod \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") "
Nov 03 21:30:08.903056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:08.903048    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-xks-ca\") pod \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\" (UID: \"8f9ca5d2-617a-4e8a-ac80-743bf430e603\") "
Nov 03 21:30:08.903432 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:08.903398    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-openshift-ca" pod "8f9ca5d2-617a-4e8a-ac80-743bf430e603" (UID: "8f9ca5d2-617a-4e8a-ac80-743bf430e603"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:08.903646 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:08.903618    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:08.905883 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:08.905854    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9ca5d2-617a-4e8a-ac80-743bf430e603-kube-api-access-4l4c2" pod "8f9ca5d2-617a-4e8a-ac80-743bf430e603" (UID: "8f9ca5d2-617a-4e8a-ac80-743bf430e603"). InnerVolumeSpecName "kube-api-access-4l4c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:08.906123 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:08.906074    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-xks-ca" pod "8f9ca5d2-617a-4e8a-ac80-743bf430e603" (UID: "8f9ca5d2-617a-4e8a-ac80-743bf430e603"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:30:09.004733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:09.004691    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l4c2\" (UniqueName: \"kubernetes.io/projected/8f9ca5d2-617a-4e8a-ac80-743bf430e603-kube-api-access-4l4c2\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:09.004733 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:09.004729    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9ca5d2-617a-4e8a-ac80-743bf430e603-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:09.639291 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:09.639259    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"
Nov 03 21:30:09.639491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:09.639303    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4" event={"ID":"8f9ca5d2-617a-4e8a-ac80-743bf430e603","Type":"ContainerDied","Data":"7c6860bfcf12cfc28198fe671c87b6bd8e8b536737a212e5822b4e0e13b911f9"}
Nov 03 21:30:09.639491 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:09.639359    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c6860bfcf12cfc28198fe671c87b6bd8e8b536737a212e5822b4e0e13b911f9"
Nov 03 21:30:11.048905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:11.048871    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"]
Nov 03 21:30:11.095869 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:11.095831    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bvmm6"]
Nov 03 21:30:12.108430 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:12.108395    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d720c369-96e9-4320-99d5-e7ac462aef76" path="/var/lib/kubelet/pods/d720c369-96e9-4320-99d5-e7ac462aef76/volumes"
Nov 03 21:30:17.590086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:17.590043    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:18.039687 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:18.039582    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:30:18.159376 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:18.159336    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:18.319949 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:18.319904    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:18.485423 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:30:18.485385    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:30:22.826870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:22.826826    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:23.025300 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:23.025244    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:26.077468 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:26.077426    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:26.245882 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:26.245834    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:28.580806 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:28.580774    5196 scope.go:122] "RemoveContainer" containerID="3f9f6059c9ea4af6d9e84744db39871457700f5e38b2eab5b9ac040031427422"
Nov 03 21:30:28.590938 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:28.590915    5196 scope.go:122] "RemoveContainer" containerID="66dc5cef0754b02d8566e99b8c4b908222e77b4198d2e5af127e597212e65985"
Nov 03 21:30:28.600840 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:28.600815    5196 scope.go:122] "RemoveContainer" containerID="88ff93008e1ea1dc39f46cd63aca76d87c0f51ae83dc9e633890b619326e611a"
Nov 03 21:30:28.930231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:28.930125    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:29.077710 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:29.077663    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:29.227418 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:29.227315    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:33.530113 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:33.530054    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:36.450620 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:36.450576    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:30:38.041739 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:38.041701    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:42.553908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:42.553869    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:47.302549 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:47.302506    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:47.628953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:47.628893    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:48.129085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:48.129047    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:48.332157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:48.332113    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:48.487889 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:30:48.487793    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:30:50.684303 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:50.684265    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:51.832684 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:51.832645    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:54.967959 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:54.967869    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:30:56.335731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:56.335700    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:56.482075 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:30:56.482040    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:00.739912 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:00.739874    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:05.228876 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:05.228835    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:09.729650 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:09.729604    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:14.255713 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:14.255668    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:18.484964 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:31:18.484923    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:31:18.743868 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:18.743776    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:22.475009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:22.474974    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:31:23.339696 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:23.339652    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:27.835294 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:27.835259    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:28.636563 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:28.636533    5196 scope.go:122] "RemoveContainer" containerID="fd904ce354892adc78690ab3c9e9d5b846d63c58e61a87f0413f9c5960f24c37"
Nov 03 21:31:28.646742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:28.646721    5196 scope.go:122] "RemoveContainer" containerID="077bca1a17e9eea0402a01d77cd5de4fadc46feca11de10db1a2778de1c84f8b"
Nov 03 21:31:28.655964 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:28.655942    5196 scope.go:122] "RemoveContainer" containerID="ef521f7ffd8b57925ae0244dfb1778bda96ad7811cae9e478aad26363015d24e"
Nov 03 21:31:32.345994 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:32.345960    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:35.624979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.624936    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"]
Nov 03 21:31:35.625558 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.625539    5196 cpu_manager.go:414] "containerMap: removing container" podUID="8f9ca5d2-617a-4e8a-ac80-743bf430e603" containerName="revoke-keys"
Nov 03 21:31:35.625640 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.625559    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9ca5d2-617a-4e8a-ac80-743bf430e603" containerName="revoke-keys"
Nov 03 21:31:35.625686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.625674    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9ca5d2-617a-4e8a-ac80-743bf430e603" containerName="revoke-keys"
Nov 03 21:31:35.630361 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.630339    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.682525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.682492    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"]
Nov 03 21:31:35.726270 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.726236    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:31:35.726270 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.726254    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:31:35.765140 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.765070    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.765363 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.765153    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.765363 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.765304    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7t6\" (UniqueName: \"kubernetes.io/projected/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-kube-api-access-mn7t6\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.866073 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.866042    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.866073 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.866084    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.866390 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.866129    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7t6\" (UniqueName: \"kubernetes.io/projected/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-kube-api-access-mn7t6\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.866742 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.866715    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.868666 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.868642    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.920394 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.920318    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7t6\" (UniqueName: \"kubernetes.io/projected/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-kube-api-access-mn7t6\") pod \"maas-api-revoke-sub-1ca3f4ce-wp6qk\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:35.944729 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:35.944697    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:36.319612 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:36.319567    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"]
Nov 03 21:31:36.327469 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:31:36.327427    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a51cd1_88d8_4b7d_8c00_a851bddfe5c9.slice/crio-67416dab8025dc0b6aaac8f9206799a58029a39370bbf31d6fcf8e32808f4104 WatchSource:0}: Error finding container 67416dab8025dc0b6aaac8f9206799a58029a39370bbf31d6fcf8e32808f4104: Status 404 returned error can't find the container with id 67416dab8025dc0b6aaac8f9206799a58029a39370bbf31d6fcf8e32808f4104
Nov 03 21:31:36.831751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:36.831716    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:37.235837 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:37.235805    5196 generic.go:356] "Generic (PLEG): container finished" podID="68a51cd1-88d8-4b7d-8c00-a851bddfe5c9" containerID="709bce97ae74ed2548cd39c977c4e4b1429aa0b4d1528227e7a926b0a60bcd0a" exitCode=0
Nov 03 21:31:37.236216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:37.235882    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk" event={"ID":"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9","Type":"ContainerDied","Data":"709bce97ae74ed2548cd39c977c4e4b1429aa0b4d1528227e7a926b0a60bcd0a"}
Nov 03 21:31:37.236216 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:37.235951    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk" event={"ID":"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9","Type":"ContainerStarted","Data":"67416dab8025dc0b6aaac8f9206799a58029a39370bbf31d6fcf8e32808f4104"}
Nov 03 21:31:38.512927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.512855    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:38.602525 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.602482    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-openshift-ca\") pod \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") "
Nov 03 21:31:38.602685 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.602562    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-kube-api-access-mn7t6\" (UniqueName: \"kubernetes.io/projected/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-kube-api-access-mn7t6\") pod \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") "
Nov 03 21:31:38.602685 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.602640    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-xks-ca\") pod \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\" (UID: \"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9\") "
Nov 03 21:31:38.602958 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.602928    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-openshift-ca" pod "68a51cd1-88d8-4b7d-8c00-a851bddfe5c9" (UID: "68a51cd1-88d8-4b7d-8c00-a851bddfe5c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:31:38.603286 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.603257    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:38.605188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.605158    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-xks-ca" pod "68a51cd1-88d8-4b7d-8c00-a851bddfe5c9" (UID: "68a51cd1-88d8-4b7d-8c00-a851bddfe5c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:31:38.605336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.605307    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-kube-api-access-mn7t6" pod "68a51cd1-88d8-4b7d-8c00-a851bddfe5c9" (UID: "68a51cd1-88d8-4b7d-8c00-a851bddfe5c9"). InnerVolumeSpecName "kube-api-access-mn7t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:31:38.704167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.704012    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn7t6\" (UniqueName: \"kubernetes.io/projected/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-kube-api-access-mn7t6\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:38.704167 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:38.704167    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:39.251084 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:39.251049    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"
Nov 03 21:31:39.251084 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:39.251075    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk" event={"ID":"68a51cd1-88d8-4b7d-8c00-a851bddfe5c9","Type":"ContainerDied","Data":"67416dab8025dc0b6aaac8f9206799a58029a39370bbf31d6fcf8e32808f4104"}
Nov 03 21:31:39.251401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:39.251144    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67416dab8025dc0b6aaac8f9206799a58029a39370bbf31d6fcf8e32808f4104"
Nov 03 21:31:40.501287 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:40.501239    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:41.345474 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:41.345436    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:45.822534 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:45.822499    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:46.693777 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:46.693731    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:47.121210 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.121169    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"]
Nov 03 21:31:47.121815 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.121798    5196 cpu_manager.go:414] "containerMap: removing container" podUID="68a51cd1-88d8-4b7d-8c00-a851bddfe5c9" containerName="revoke-keys"
Nov 03 21:31:47.121865 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.121816    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a51cd1-88d8-4b7d-8c00-a851bddfe5c9" containerName="revoke-keys"
Nov 03 21:31:47.121901 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.121895    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a51cd1-88d8-4b7d-8c00-a851bddfe5c9" containerName="revoke-keys"
Nov 03 21:31:47.125573 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.125550    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.177827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.177792    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"]
Nov 03 21:31:47.197931 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.197897    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cklpk\" (UniqueName: \"kubernetes.io/projected/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-kube-api-access-cklpk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.198123 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.197940    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.198123 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.197973    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.221056 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.221022    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:31:47.221259 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.221179    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vltbh\""
Nov 03 21:31:47.299338 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.299302    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cklpk\" (UniqueName: \"kubernetes.io/projected/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-kube-api-access-cklpk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.299495 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.299345    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.299495 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.299395    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.300405 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.300323    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.302292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.302262    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.353260 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.353224    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cklpk\" (UniqueName: \"kubernetes.io/projected/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-kube-api-access-cklpk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.439979 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.439870    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:47.815999 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:47.815918    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"]
Nov 03 21:31:47.822993 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:31:47.822947    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2390e7dd_e353_49c8_97e2_4d98ceeb38fa.slice/crio-765b12539fac287c79460db70d6e8a0a353c0af8f49040e14f09032ccf6af930 WatchSource:0}: Error finding container 765b12539fac287c79460db70d6e8a0a353c0af8f49040e14f09032ccf6af930: Status 404 returned error can't find the container with id 765b12539fac287c79460db70d6e8a0a353c0af8f49040e14f09032ccf6af930
Nov 03 21:31:48.314603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:48.314556    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt" event={"ID":"2390e7dd-e353-49c8-97e2-4d98ceeb38fa","Type":"ContainerStarted","Data":"95a8cb0ef0300cbabdb9eb884d3768dee41e26aea5546f95f0c5ea631289dc91"}
Nov 03 21:31:48.315088 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:48.314619    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt" event={"ID":"2390e7dd-e353-49c8-97e2-4d98ceeb38fa","Type":"ContainerStarted","Data":"765b12539fac287c79460db70d6e8a0a353c0af8f49040e14f09032ccf6af930"}
Nov 03 21:31:48.423487 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:48.423426    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt" podStartSLOduration=1.4234130409999999 podStartE2EDuration="1.423413041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:48.420808106 +0000 UTC m=+3748.424766853" watchObservedRunningTime="2026-11-03 21:31:48.423413041 +0000 UTC m=+3748.427371764"
Nov 03 21:31:48.485839 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:31:48.485803    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:31:48.544812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:48.544732    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:31:49.322996 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:49.322959    5196 generic.go:356] "Generic (PLEG): container finished" podID="2390e7dd-e353-49c8-97e2-4d98ceeb38fa" containerID="95a8cb0ef0300cbabdb9eb884d3768dee41e26aea5546f95f0c5ea631289dc91" exitCode=0
Nov 03 21:31:49.323405 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:49.323047    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt" event={"ID":"2390e7dd-e353-49c8-97e2-4d98ceeb38fa","Type":"ContainerDied","Data":"95a8cb0ef0300cbabdb9eb884d3768dee41e26aea5546f95f0c5ea631289dc91"}
Nov 03 21:31:50.484451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.484422    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:50.537644 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.537603    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:50.538615 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.538595    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-kube-api-access-cklpk\" (UniqueName: \"kubernetes.io/projected/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-kube-api-access-cklpk\") pod \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") "
Nov 03 21:31:50.538688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.538654    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-openshift-ca\") pod \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") "
Nov 03 21:31:50.538741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.538710    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-xks-ca\") pod \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\" (UID: \"2390e7dd-e353-49c8-97e2-4d98ceeb38fa\") "
Nov 03 21:31:50.539039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.539015    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-openshift-ca" pod "2390e7dd-e353-49c8-97e2-4d98ceeb38fa" (UID: "2390e7dd-e353-49c8-97e2-4d98ceeb38fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:31:50.541015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.540988    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-xks-ca" pod "2390e7dd-e353-49c8-97e2-4d98ceeb38fa" (UID: "2390e7dd-e353-49c8-97e2-4d98ceeb38fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:31:50.541015 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.541005    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-kube-api-access-cklpk" pod "2390e7dd-e353-49c8-97e2-4d98ceeb38fa" (UID: "2390e7dd-e353-49c8-97e2-4d98ceeb38fa"). InnerVolumeSpecName "kube-api-access-cklpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:31:50.640484 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.640389    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cklpk\" (UniqueName: \"kubernetes.io/projected/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-kube-api-access-cklpk\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:50.640484 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.640425    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:50.640484 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:50.640438    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2390e7dd-e353-49c8-97e2-4d98ceeb38fa-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:51.337831 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:51.337791    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt" event={"ID":"2390e7dd-e353-49c8-97e2-4d98ceeb38fa","Type":"ContainerDied","Data":"765b12539fac287c79460db70d6e8a0a353c0af8f49040e14f09032ccf6af930"}
Nov 03 21:31:51.338024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:51.337841    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="765b12539fac287c79460db70d6e8a0a353c0af8f49040e14f09032ccf6af930"
Nov 03 21:31:51.338024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:51.337808    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"
Nov 03 21:31:51.727910 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:51.727810    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:55.133163 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:55.133120    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:56.753343 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:56.753305    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:58.981482 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:58.981442    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:59.128727 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:31:59.128624    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:01.044759 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:01.044722    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"]
Nov 03 21:32:01.091309 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:01.091264    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40786756-8rvlc"]
Nov 03 21:32:02.108953 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:02.108916    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da152cc-5f4e-40c3-80df-d9237dd1eda0" path="/var/lib/kubelet/pods/6da152cc-5f4e-40c3-80df-d9237dd1eda0/volumes"
Nov 03 21:32:03.000258 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:03.000216    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:03.253839 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:03.253735    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:03.573780 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:03.573742    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:32:03.620877 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:03.620840    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"]
Nov 03 21:32:03.623534 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:03.623508    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6d8154-xtfzt"]
Nov 03 21:32:03.981260 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:03.981158    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:04.109030 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:04.108992    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2390e7dd-e353-49c8-97e2-4d98ceeb38fa" path="/var/lib/kubelet/pods/2390e7dd-e353-49c8-97e2-4d98ceeb38fa/volumes"
Nov 03 21:32:10.032316 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:10.032276    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:10.191718 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:10.191682    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:10.341948 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:10.341911    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:17.159938 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:17.159894    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:18.485804 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:32:18.485765    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:32:21.342541 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:21.342500    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:32:22.438028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:22.437962    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:22.615312 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:22.615269    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:27.766688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:27.766642    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:28.781366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:28.781325    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:42.722553 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:42.722514    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:47.049497 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:47.049464    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"]
Nov 03 21:32:47.096243 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:47.096212    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76d9l"]
Nov 03 21:32:48.108071 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:48.108034    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae2065e-32f7-4c3d-9c13-cedd97e0d79e" path="/var/lib/kubelet/pods/bae2065e-32f7-4c3d-9c13-cedd97e0d79e/volumes"
Nov 03 21:32:48.485815 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:32:48.485718    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:32:57.100532 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:57.100495    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:57.301031 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:57.300993    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:57.740632 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:32:57.740596    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:33:02.084161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:02.084122    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:33:05.369120 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:05.368052    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:33:18.485732 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:33:18.485687    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:33:28.718366 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:28.718329    5196 scope.go:122] "RemoveContainer" containerID="9d128d603283778c5f8bf88e8fc83f7c216a30de1790ab957638281e6154bd8f"
Nov 03 21:33:28.728499 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:28.728471    5196 scope.go:122] "RemoveContainer" containerID="8ca43bedd57b53492ec43228a476679287126333a774878a37b2a20b4a2e7859"
Nov 03 21:33:29.159751 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:29.159706    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:33:40.766596 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:40.766547    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:33:44.969502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:44.969452    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:33:48.485761 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:33:48.485711    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:33:57.407029 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:57.406989    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:33:59.748189 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:33:59.748147    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:01.308879 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:01.308833    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:03.935086 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:03.935053    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:34:16.547305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:16.547267    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:34:18.530903 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:34:18.530865    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:34:20.850745 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:20.850713    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:34:20.857560 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:20.857535    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:34:20.873128 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:20.873073    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:34:20.879759 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:20.879735    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:34:26.784625 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:26.784588    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:34:28.762886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:28.762853    5196 scope.go:122] "RemoveContainer" containerID="4d56474102cf2d899bc76f132be541b5352b636b951e68071511fdd601489f74"
Nov 03 21:34:31.639570 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:31.639528    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:31.806005 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:31.805967    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:47.073864 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:47.073820    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:34:48.486110 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:34:48.486047    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:34:53.936741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:53.936700    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:54.646328 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:54.646290    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:54.879560 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:54.879525    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:57.735719 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:57.735672    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:57.899683 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:34:57.899644    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:00.866723 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:00.866680    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:01.063268 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:01.063230    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:01.915178 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:01.915143    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:02.071731 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:02.071674    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:02.219660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:02.219576    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:04.024660 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:04.024621    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:04.168296 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:04.168255    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:06.930028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:06.929992    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:09.043538 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:09.043497    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"]
Nov 03 21:35:09.090186 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:09.090139    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zzpp4"]
Nov 03 21:35:09.666519 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:09.666484    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:35:10.110230 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:10.110183    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f9ca5d2-617a-4e8a-ac80-743bf430e603" path="/var/lib/kubelet/pods/8f9ca5d2-617a-4e8a-ac80-743bf430e603/volumes"
Nov 03 21:35:11.535219 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:11.535180    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:16.033716 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:16.033679    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:18.485827 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:35:18.485791    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:35:20.739752 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:20.739717    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:25.151203 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:25.151116    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:29.884925 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:29.884886    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:34.547800 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:34.547762    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:38.832614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.832581    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"]
Nov 03 21:35:38.833378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.833358    5196 cpu_manager.go:414] "containerMap: removing container" podUID="2390e7dd-e353-49c8-97e2-4d98ceeb38fa" containerName="revoke-keys"
Nov 03 21:35:38.833489 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.833379    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2390e7dd-e353-49c8-97e2-4d98ceeb38fa" containerName="revoke-keys"
Nov 03 21:35:38.833548 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.833514    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="2390e7dd-e353-49c8-97e2-4d98ceeb38fa" containerName="revoke-keys"
Nov 03 21:35:38.836769 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.836748    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:38.890372 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.890335    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"]
Nov 03 21:35:38.906423 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.906378    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pgv\" (UniqueName: \"kubernetes.io/projected/08e4ea52-57ef-44ba-b9a6-22508ea4f082-kube-api-access-79pgv\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:38.906648 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.906490    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:38.906648 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.906606    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:38.933457 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.933418    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:35:38.933646 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:38.933466    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tssvb\""
Nov 03 21:35:39.007767 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.007732    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pgv\" (UniqueName: \"kubernetes.io/projected/08e4ea52-57ef-44ba-b9a6-22508ea4f082-kube-api-access-79pgv\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:39.007976 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.007786    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:39.007976 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.007841    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:39.008614 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.008594    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:39.010439 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.010416    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:39.035750 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.035710    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:39.063860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.063789    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pgv\" (UniqueName: \"kubernetes.io/projected/08e4ea52-57ef-44ba-b9a6-22508ea4f082-kube-api-access-79pgv\") pod \"maas-api-revoke-keys-e2e-ait-62df72ff-zdzql\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:39.151364 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.151259    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:39.526561 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:39.526459    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"]
Nov 03 21:35:39.537075 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:35:39.537036    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e4ea52_57ef_44ba_b9a6_22508ea4f082.slice/crio-97a3013931234426a71bb773f9e6ff03d2a3af433c05311132204d08972ad7a2 WatchSource:0}: Error finding container 97a3013931234426a71bb773f9e6ff03d2a3af433c05311132204d08972ad7a2: Status 404 returned error can't find the container with id 97a3013931234426a71bb773f9e6ff03d2a3af433c05311132204d08972ad7a2
Nov 03 21:35:39.740751 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:35:39.740713    5196 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e4ea52_57ef_44ba_b9a6_22508ea4f082.slice/crio-db740c5d0e1ff3d5122c2d4dc2a09abfbfb555855a0ba9560671b265cb3054ff.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 21:35:40.080054 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:40.080013    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql" event={"ID":"08e4ea52-57ef-44ba-b9a6-22508ea4f082","Type":"ContainerStarted","Data":"db740c5d0e1ff3d5122c2d4dc2a09abfbfb555855a0ba9560671b265cb3054ff"}
Nov 03 21:35:40.080514 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:40.080078    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql" event={"ID":"08e4ea52-57ef-44ba-b9a6-22508ea4f082","Type":"ContainerStarted","Data":"97a3013931234426a71bb773f9e6ff03d2a3af433c05311132204d08972ad7a2"}
Nov 03 21:35:40.199335 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:40.199219    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql" podStartSLOduration=2.199188571 podStartE2EDuration="2.199188571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:40.186918132 +0000 UTC m=+3980.190876872" watchObservedRunningTime="2026-11-03 21:35:40.199188571 +0000 UTC m=+3980.203147299"
Nov 03 21:35:41.088941 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:41.088902    5196 generic.go:356] "Generic (PLEG): container finished" podID="08e4ea52-57ef-44ba-b9a6-22508ea4f082" containerID="db740c5d0e1ff3d5122c2d4dc2a09abfbfb555855a0ba9560671b265cb3054ff" exitCode=0
Nov 03 21:35:41.089413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:41.088986    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql" event={"ID":"08e4ea52-57ef-44ba-b9a6-22508ea4f082","Type":"ContainerDied","Data":"db740c5d0e1ff3d5122c2d4dc2a09abfbfb555855a0ba9560671b265cb3054ff"}
Nov 03 21:35:42.247033 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.247004    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:42.342077 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.342039    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e4ea52-57ef-44ba-b9a6-22508ea4f082-kube-api-access-79pgv\" (UniqueName: \"kubernetes.io/projected/08e4ea52-57ef-44ba-b9a6-22508ea4f082-kube-api-access-79pgv\") pod \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") "
Nov 03 21:35:42.342306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.342159    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-xks-ca\") pod \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") "
Nov 03 21:35:42.342306 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.342282    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-openshift-ca\") pod \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\" (UID: \"08e4ea52-57ef-44ba-b9a6-22508ea4f082\") "
Nov 03 21:35:42.342730 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.342700    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-openshift-ca" pod "08e4ea52-57ef-44ba-b9a6-22508ea4f082" (UID: "08e4ea52-57ef-44ba-b9a6-22508ea4f082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:42.344471 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.344449    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-xks-ca" pod "08e4ea52-57ef-44ba-b9a6-22508ea4f082" (UID: "08e4ea52-57ef-44ba-b9a6-22508ea4f082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:35:42.344581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.344556    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e4ea52-57ef-44ba-b9a6-22508ea4f082-kube-api-access-79pgv" pod "08e4ea52-57ef-44ba-b9a6-22508ea4f082" (UID: "08e4ea52-57ef-44ba-b9a6-22508ea4f082"). InnerVolumeSpecName "kube-api-access-79pgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:42.443339 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.443302    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79pgv\" (UniqueName: \"kubernetes.io/projected/08e4ea52-57ef-44ba-b9a6-22508ea4f082-kube-api-access-79pgv\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:42.443339 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.443337    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:42.443576 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:42.443351    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e4ea52-57ef-44ba-b9a6-22508ea4f082-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:43.104864 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:43.104832    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"
Nov 03 21:35:43.104864 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:43.104857    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql" event={"ID":"08e4ea52-57ef-44ba-b9a6-22508ea4f082","Type":"ContainerDied","Data":"97a3013931234426a71bb773f9e6ff03d2a3af433c05311132204d08972ad7a2"}
Nov 03 21:35:43.105065 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:43.104901    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97a3013931234426a71bb773f9e6ff03d2a3af433c05311132204d08972ad7a2"
Nov 03 21:35:43.438053 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:43.437953    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:47.925659 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:47.925619    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:48.484753 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:35:48.484716    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:35:50.658481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:50.658442    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:50.852757 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:50.852712    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:52.345812 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:52.345768    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:56.251446 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:56.251403    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:57.719009 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:57.718972    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:57.980561 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:57.980478    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:58.133508 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:58.133471    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:58.158579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:58.158542    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"]
Nov 03 21:35:58.205545 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:58.205507    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62df72ff-zdzql"]
Nov 03 21:35:58.349149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:58.349080    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:59.890508 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:35:59.890460    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:00.109034 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:00.108988    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e4ea52-57ef-44ba-b9a6-22508ea4f082" path="/var/lib/kubelet/pods/08e4ea52-57ef-44ba-b9a6-22508ea4f082/volumes"
Nov 03 21:36:00.654446 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:00.654409    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:36:01.052019 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:01.051930    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:01.616995 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:01.616957    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:01.897309 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:01.897208    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:05.487072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:05.487022    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:05.659237 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:05.659196    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:05.815190 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:05.815085    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:08.681445 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:08.681411    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:08.877310 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:08.877268    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:11.695260 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:11.695222    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:11.887401 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:11.887364    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:16.324465 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:16.324427    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:18.486158 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:36:18.486113    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:36:19.179150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:19.179111    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:36:20.837476 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:20.837437    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:25.328359 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:25.328321    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:28.803048 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:28.803008    5196 scope.go:122] "RemoveContainer" containerID="8e198f302c2b4c1259658ba4addab33b504ab43cc17dfd5c1f0f5d2e599198e5"
Nov 03 21:36:29.941784 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:29.941740    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:30.252647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:30.252553    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:34.449475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:34.449437    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:38.938595 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:38.938554    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:39.044358 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:39.044312    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"]
Nov 03 21:36:39.046669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:39.046643    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca3f4ce-wp6qk"]
Nov 03 21:36:40.109129 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:40.109077    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a51cd1-88d8-4b7d-8c00-a851bddfe5c9" path="/var/lib/kubelet/pods/68a51cd1-88d8-4b7d-8c00-a851bddfe5c9/volumes"
Nov 03 21:36:43.346417 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:43.346376    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:47.834029 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:47.833996    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:48.485632 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:36:48.485597    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:36:52.334553 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:52.334519    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:56.831924 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:56.831839    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:58.401721 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:36:58.401676    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:01.333741 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:01.333703    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:05.850985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:05.850952    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:10.341065 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:10.341010    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:14.838305 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:14.838264    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:18.486745 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:37:18.486705    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:37:19.318327 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:19.318289    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:23.835581 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:23.835540    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:26.702991 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:26.701124    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:26.889514 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:26.889468    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:28.324940 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:28.324886    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:32.878321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:32.878273    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:37.319925 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:37.319888    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:41.877631 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:41.877595    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:46.332950 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:46.332905    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:48.485360 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:37:48.485319    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:37:50.836640 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:50.836578    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:53.173635 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.173599    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"]
Nov 03 21:37:53.174189 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.174173    5196 cpu_manager.go:414] "containerMap: removing container" podUID="08e4ea52-57ef-44ba-b9a6-22508ea4f082" containerName="revoke-keys"
Nov 03 21:37:53.174254 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.174190    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e4ea52-57ef-44ba-b9a6-22508ea4f082" containerName="revoke-keys"
Nov 03 21:37:53.174321 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.174309    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e4ea52-57ef-44ba-b9a6-22508ea4f082" containerName="revoke-keys"
Nov 03 21:37:53.177611 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.177591    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.230780 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.230740    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"]
Nov 03 21:37:53.274106 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.274058    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:37:53.274286 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.274136    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tssvb\""
Nov 03 21:37:53.276633 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.276609    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p97v\" (UniqueName: \"kubernetes.io/projected/520ff895-9829-424f-875f-b65381a21b92-kube-api-access-9p97v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.276692 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.276675    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520ff895-9829-424f-875f-b65381a21b92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.276803 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.276779    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520ff895-9829-424f-875f-b65381a21b92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.377907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.377868    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p97v\" (UniqueName: \"kubernetes.io/projected/520ff895-9829-424f-875f-b65381a21b92-kube-api-access-9p97v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.378132 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.377931    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520ff895-9829-424f-875f-b65381a21b92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.378132 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.377964    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520ff895-9829-424f-875f-b65381a21b92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.378591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.378570    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520ff895-9829-424f-875f-b65381a21b92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.380499 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.380478    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520ff895-9829-424f-875f-b65381a21b92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.431849 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.431773    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p97v\" (UniqueName: \"kubernetes.io/projected/520ff895-9829-424f-875f-b65381a21b92-kube-api-access-9p97v\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.491443 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.491408    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:53.866299 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.866265    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"]
Nov 03 21:37:53.873288 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:37:53.873250    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520ff895_9829_424f_875f_b65381a21b92.slice/crio-1a4665d2b6fb7865584a2db5953628c43e3f245c349753dd28e9fa71cfe3f233 WatchSource:0}: Error finding container 1a4665d2b6fb7865584a2db5953628c43e3f245c349753dd28e9fa71cfe3f233: Status 404 returned error can't find the container with id 1a4665d2b6fb7865584a2db5953628c43e3f245c349753dd28e9fa71cfe3f233
Nov 03 21:37:53.970533 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:53.970495    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m" event={"ID":"520ff895-9829-424f-875f-b65381a21b92","Type":"ContainerStarted","Data":"1a4665d2b6fb7865584a2db5953628c43e3f245c349753dd28e9fa71cfe3f233"}
Nov 03 21:37:54.977926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:54.977891    5196 generic.go:356] "Generic (PLEG): container finished" podID="520ff895-9829-424f-875f-b65381a21b92" containerID="1cd94c5ec0b56e94cd1f68ef6da86964ebff35e743141163adf8e2cf78ee6b81" exitCode=0
Nov 03 21:37:54.978386 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:54.977988    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m" event={"ID":"520ff895-9829-424f-875f-b65381a21b92","Type":"ContainerDied","Data":"1cd94c5ec0b56e94cd1f68ef6da86964ebff35e743141163adf8e2cf78ee6b81"}
Nov 03 21:37:55.488431 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:55.488398    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:56.136528 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.136502    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:56.309457 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.309363    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520ff895-9829-424f-875f-b65381a21b92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520ff895-9829-424f-875f-b65381a21b92-maas-api-xks-ca\") pod \"520ff895-9829-424f-875f-b65381a21b92\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") "
Nov 03 21:37:56.309647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.309480    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520ff895-9829-424f-875f-b65381a21b92-kube-api-access-9p97v\" (UniqueName: \"kubernetes.io/projected/520ff895-9829-424f-875f-b65381a21b92-kube-api-access-9p97v\") pod \"520ff895-9829-424f-875f-b65381a21b92\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") "
Nov 03 21:37:56.309647 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.309545    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520ff895-9829-424f-875f-b65381a21b92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520ff895-9829-424f-875f-b65381a21b92-maas-api-openshift-ca\") pod \"520ff895-9829-424f-875f-b65381a21b92\" (UID: \"520ff895-9829-424f-875f-b65381a21b92\") "
Nov 03 21:37:56.309878 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.309853    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520ff895-9829-424f-875f-b65381a21b92-maas-api-openshift-ca" pod "520ff895-9829-424f-875f-b65381a21b92" (UID: "520ff895-9829-424f-875f-b65381a21b92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:37:56.311775 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.311753    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520ff895-9829-424f-875f-b65381a21b92-kube-api-access-9p97v" pod "520ff895-9829-424f-875f-b65381a21b92" (UID: "520ff895-9829-424f-875f-b65381a21b92"). InnerVolumeSpecName "kube-api-access-9p97v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:56.311871 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.311836    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520ff895-9829-424f-875f-b65381a21b92-maas-api-xks-ca" pod "520ff895-9829-424f-875f-b65381a21b92" (UID: "520ff895-9829-424f-875f-b65381a21b92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:37:56.411251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.411214    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/520ff895-9829-424f-875f-b65381a21b92-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:56.411251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.411245    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/520ff895-9829-424f-875f-b65381a21b92-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:56.411251 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:56.411254    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p97v\" (UniqueName: \"kubernetes.io/projected/520ff895-9829-424f-875f-b65381a21b92-kube-api-access-9p97v\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:57.001262 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:57.001218    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"
Nov 03 21:37:57.001467 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:57.001250    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m" event={"ID":"520ff895-9829-424f-875f-b65381a21b92","Type":"ContainerDied","Data":"1a4665d2b6fb7865584a2db5953628c43e3f245c349753dd28e9fa71cfe3f233"}
Nov 03 21:37:57.001467 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:57.001352    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4665d2b6fb7865584a2db5953628c43e3f245c349753dd28e9fa71cfe3f233"
Nov 03 21:37:58.718935 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:37:58.718898    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:00.231862 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:00.231829    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:04.644045 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:04.644004    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:05.088592 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:05.088559    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:05.232949 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:05.232901    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:13.639181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:13.639140    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:15.039909 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:15.039869    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:15.300411 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:15.300311    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:15.590076 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:15.590044    5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"]
Nov 03 21:38:15.636304 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:15.636268    5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1850fc-wp87m"]
Nov 03 21:38:16.005495 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:16.005406    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:16.108947 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:16.108910    5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520ff895-9829-424f-875f-b65381a21b92" path="/var/lib/kubelet/pods/520ff895-9829-424f-875f-b65381a21b92/volumes"
Nov 03 21:38:17.961454 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:17.961420    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:38:18.482754 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:38:18.482718    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:38:21.532455 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:21.532422    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:21.727735 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:21.727701    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:21.884161 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:21.884127    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:28.841071 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:28.841039    5196 scope.go:122] "RemoveContainer" containerID="95a8cb0ef0300cbabdb9eb884d3768dee41e26aea5546f95f0c5ea631289dc91"
Nov 03 21:38:28.851254 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:28.851233    5196 scope.go:122] "RemoveContainer" containerID="709bce97ae74ed2548cd39c977c4e4b1429aa0b4d1528227e7a926b0a60bcd0a"
Nov 03 21:38:29.129392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:29.129284    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:38:40.579425 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:40.579386    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:38:41.776257 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:41.776216    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:43.058253 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:43.058208    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fm4zc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:38:43.058905 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:43.058870    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fm4zc" podUID="1cba3ff6-170a-45dc-8cf7-1517813237c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:38:43.325284 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:43.325236    5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-tcqsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:38:43.325503 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:43.325331    5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-tcqsm" podUID="3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:38:48.493642 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:38:48.493601    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:38:48.785148 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.785036    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"]
Nov 03 21:38:48.785688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.785660    5196 cpu_manager.go:414] "containerMap: removing container" podUID="520ff895-9829-424f-875f-b65381a21b92" containerName="revoke-keys"
Nov 03 21:38:48.785688 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.785688    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520ff895-9829-424f-875f-b65381a21b92" containerName="revoke-keys"
Nov 03 21:38:48.785868 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.785829    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="520ff895-9829-424f-875f-b65381a21b92" containerName="revoke-keys"
Nov 03 21:38:48.790836 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.790815    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:48.841674 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.841636    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"]
Nov 03 21:38:48.884894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.884855    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:38:48.884894 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.884865    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tssvb\""
Nov 03 21:38:48.953276 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.953237    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vphr8\" (UniqueName: \"kubernetes.io/projected/60ff5464-5661-479f-a4fb-3e0368f0979f-kube-api-access-vphr8\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:48.953475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.953405    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:48.953475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:48.953443    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:49.054651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:49.054568    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:49.054651 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:49.054610    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:49.054881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:49.054677    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vphr8\" (UniqueName: \"kubernetes.io/projected/60ff5464-5661-479f-a4fb-3e0368f0979f-kube-api-access-vphr8\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:49.055340 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:49.055313    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:49.057085 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:49.057055    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:49.109920 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:49.109883    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vphr8\" (UniqueName: \"kubernetes.io/projected/60ff5464-5661-479f-a4fb-3e0368f0979f-kube-api-access-vphr8\") pod \"maas-api-revoke-sub-f03629db-zjp9m\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:49.405181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:49.405140    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:49.778345 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:49.778255    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"]
Nov 03 21:38:49.785568 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:38:49.785535    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ff5464_5661_479f_a4fb_3e0368f0979f.slice/crio-afc97bc2f9277923e4be192d7fa4bb45bf255204380c32660ab47174a6171d32 WatchSource:0}: Error finding container afc97bc2f9277923e4be192d7fa4bb45bf255204380c32660ab47174a6171d32: Status 404 returned error can't find the container with id afc97bc2f9277923e4be192d7fa4bb45bf255204380c32660ab47174a6171d32
Nov 03 21:38:50.375687 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:50.375653    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m" event={"ID":"60ff5464-5661-479f-a4fb-3e0368f0979f","Type":"ContainerStarted","Data":"3fc7237513176a017318ed96ffb908706cb7b99870b06dbb04bb8cbf263fcbc5"}
Nov 03 21:38:50.375886 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:50.375704    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m" event={"ID":"60ff5464-5661-479f-a4fb-3e0368f0979f","Type":"ContainerStarted","Data":"afc97bc2f9277923e4be192d7fa4bb45bf255204380c32660ab47174a6171d32"}
Nov 03 21:38:50.482791 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:50.482740    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m" podStartSLOduration=2.482726583 podStartE2EDuration="2.482726583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:50.482055648 +0000 UTC m=+4170.486014397" watchObservedRunningTime="2026-11-03 21:38:50.482726583 +0000 UTC m=+4170.486685306"
Nov 03 21:38:51.384039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:51.384006    5196 generic.go:356] "Generic (PLEG): container finished" podID="60ff5464-5661-479f-a4fb-3e0368f0979f" containerID="3fc7237513176a017318ed96ffb908706cb7b99870b06dbb04bb8cbf263fcbc5" exitCode=0
Nov 03 21:38:51.384511 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:51.384121    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m" event={"ID":"60ff5464-5661-479f-a4fb-3e0368f0979f","Type":"ContainerDied","Data":"3fc7237513176a017318ed96ffb908706cb7b99870b06dbb04bb8cbf263fcbc5"}
Nov 03 21:38:52.338173 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.338136    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:52.532285 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.532257    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:52.695870 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.695766    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ff5464-5661-479f-a4fb-3e0368f0979f-kube-api-access-vphr8\" (UniqueName: \"kubernetes.io/projected/60ff5464-5661-479f-a4fb-3e0368f0979f-kube-api-access-vphr8\") pod \"60ff5464-5661-479f-a4fb-3e0368f0979f\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") "
Nov 03 21:38:52.696044 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.695891    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-openshift-ca\") pod \"60ff5464-5661-479f-a4fb-3e0368f0979f\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") "
Nov 03 21:38:52.696044 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.695987    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-xks-ca\") pod \"60ff5464-5661-479f-a4fb-3e0368f0979f\" (UID: \"60ff5464-5661-479f-a4fb-3e0368f0979f\") "
Nov 03 21:38:52.696349 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.696321    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-openshift-ca" pod "60ff5464-5661-479f-a4fb-3e0368f0979f" (UID: "60ff5464-5661-479f-a4fb-3e0368f0979f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:38:52.698292 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.698266    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-xks-ca" pod "60ff5464-5661-479f-a4fb-3e0368f0979f" (UID: "60ff5464-5661-479f-a4fb-3e0368f0979f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:38:52.698405 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.698325    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ff5464-5661-479f-a4fb-3e0368f0979f-kube-api-access-vphr8" pod "60ff5464-5661-479f-a4fb-3e0368f0979f" (UID: "60ff5464-5661-479f-a4fb-3e0368f0979f"). InnerVolumeSpecName "kube-api-access-vphr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:38:52.797026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.796993    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:52.797026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.797023    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ff5464-5661-479f-a4fb-3e0368f0979f-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:52.797026 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:52.797033    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vphr8\" (UniqueName: \"kubernetes.io/projected/60ff5464-5661-479f-a4fb-3e0368f0979f-kube-api-access-vphr8\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:53.398724 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:53.398693    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m"
Nov 03 21:38:53.398927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:53.398730    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zjp9m" event={"ID":"60ff5464-5661-479f-a4fb-3e0368f0979f","Type":"ContainerDied","Data":"afc97bc2f9277923e4be192d7fa4bb45bf255204380c32660ab47174a6171d32"}
Nov 03 21:38:53.398927 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:53.398783    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc97bc2f9277923e4be192d7fa4bb45bf255204380c32660ab47174a6171d32"
Nov 03 21:38:59.047451 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:59.047409    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:59.333044 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:38:59.332998    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:39:01.461502 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:01.461439    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:39:09.474017 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.473978    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"]
Nov 03 21:39:09.474540 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.474525    5196 cpu_manager.go:414] "containerMap: removing container" podUID="60ff5464-5661-479f-a4fb-3e0368f0979f" containerName="revoke-keys"
Nov 03 21:39:09.474599 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.474542    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ff5464-5661-479f-a4fb-3e0368f0979f" containerName="revoke-keys"
Nov 03 21:39:09.474652 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.474643    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ff5464-5661-479f-a4fb-3e0368f0979f" containerName="revoke-keys"
Nov 03 21:39:09.477926 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.477906    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.532018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.531986    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"]
Nov 03 21:39:09.575125 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.575080    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:39:09.575300 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.575165    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tssvb\""
Nov 03 21:39:09.631827 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.631793    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.631985 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.631946    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.632047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.631992    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd4f6\" (UniqueName: \"kubernetes.io/projected/78bd51b9-267e-48d0-9f6d-8831e5201fb2-kube-api-access-zd4f6\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.732779 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.732677    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.732779 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.732718    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd4f6\" (UniqueName: \"kubernetes.io/projected/78bd51b9-267e-48d0-9f6d-8831e5201fb2-kube-api-access-zd4f6\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.733027 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.732803    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.733553 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.733531    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.735428 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.735406    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.786707 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.786665    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd4f6\" (UniqueName: \"kubernetes.io/projected/78bd51b9-267e-48d0-9f6d-8831e5201fb2-kube-api-access-zd4f6\") pod \"maas-api-revoke-sub-0165d7d2-t7c66\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:09.791309 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:09.791278    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:10.167667 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:10.167631    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"]
Nov 03 21:39:10.175568 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:39:10.175531    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bd51b9_267e_48d0_9f6d_8831e5201fb2.slice/crio-ef3bd430637dc0c6f0e86b5b5a912dd88f5357a13ab82d7aa5cb59f26caabe09 WatchSource:0}: Error finding container ef3bd430637dc0c6f0e86b5b5a912dd88f5357a13ab82d7aa5cb59f26caabe09: Status 404 returned error can't find the container with id ef3bd430637dc0c6f0e86b5b5a912dd88f5357a13ab82d7aa5cb59f26caabe09
Nov 03 21:39:10.535419 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:10.535378    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66" event={"ID":"78bd51b9-267e-48d0-9f6d-8831e5201fb2","Type":"ContainerStarted","Data":"64b53034283dec2c6252d450a03298fc7035e7019f59372acd9008a4e0c1547d"}
Nov 03 21:39:10.535932 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:10.535443    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66" event={"ID":"78bd51b9-267e-48d0-9f6d-8831e5201fb2","Type":"ContainerStarted","Data":"ef3bd430637dc0c6f0e86b5b5a912dd88f5357a13ab82d7aa5cb59f26caabe09"}
Nov 03 21:39:10.646112 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:10.646026    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66" podStartSLOduration=1.646011638 podStartE2EDuration="1.646011638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:10.644358617 +0000 UTC m=+4190.648317362" watchObservedRunningTime="2026-11-03 21:39:10.646011638 +0000 UTC m=+4190.649970362"
Nov 03 21:39:11.543188 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:11.543148    5196 generic.go:356] "Generic (PLEG): container finished" podID="78bd51b9-267e-48d0-9f6d-8831e5201fb2" containerID="64b53034283dec2c6252d450a03298fc7035e7019f59372acd9008a4e0c1547d" exitCode=0
Nov 03 21:39:11.543610 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:11.543232    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66" event={"ID":"78bd51b9-267e-48d0-9f6d-8831e5201fb2","Type":"ContainerDied","Data":"64b53034283dec2c6252d450a03298fc7035e7019f59372acd9008a4e0c1547d"}
Nov 03 21:39:12.707658 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.707628    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:12.762909 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.762858    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-openshift-ca\") pod \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") "
Nov 03 21:39:12.763164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.762981    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78bd51b9-267e-48d0-9f6d-8831e5201fb2-kube-api-access-zd4f6\" (UniqueName: \"kubernetes.io/projected/78bd51b9-267e-48d0-9f6d-8831e5201fb2-kube-api-access-zd4f6\") pod \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") "
Nov 03 21:39:12.763164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.763035    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-xks-ca\") pod \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\" (UID: \"78bd51b9-267e-48d0-9f6d-8831e5201fb2\") "
Nov 03 21:39:12.763308 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.763266    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-openshift-ca" pod "78bd51b9-267e-48d0-9f6d-8831e5201fb2" (UID: "78bd51b9-267e-48d0-9f6d-8831e5201fb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:12.763475 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.763454    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:12.765408 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.765378    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-xks-ca" pod "78bd51b9-267e-48d0-9f6d-8831e5201fb2" (UID: "78bd51b9-267e-48d0-9f6d-8831e5201fb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:12.765527 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.765480    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bd51b9-267e-48d0-9f6d-8831e5201fb2-kube-api-access-zd4f6" pod "78bd51b9-267e-48d0-9f6d-8831e5201fb2" (UID: "78bd51b9-267e-48d0-9f6d-8831e5201fb2"). InnerVolumeSpecName "kube-api-access-zd4f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:12.864379 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.864323    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd4f6\" (UniqueName: \"kubernetes.io/projected/78bd51b9-267e-48d0-9f6d-8831e5201fb2-kube-api-access-zd4f6\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:12.864379 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:12.864371    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78bd51b9-267e-48d0-9f6d-8831e5201fb2-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:13.557119 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:13.557057    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66"
Nov 03 21:39:13.557326 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:13.557115    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-t7c66" event={"ID":"78bd51b9-267e-48d0-9f6d-8831e5201fb2","Type":"ContainerDied","Data":"ef3bd430637dc0c6f0e86b5b5a912dd88f5357a13ab82d7aa5cb59f26caabe09"}
Nov 03 21:39:13.557326 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:13.557170    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef3bd430637dc0c6f0e86b5b5a912dd88f5357a13ab82d7aa5cb59f26caabe09"
Nov 03 21:39:18.485076 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:39:18.485035    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:39:20.897102 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:20.897059    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:39:20.903642 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:20.903618    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:39:20.932006 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:20.931977    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v455q_170428c7-02e4-40f5-a88c-1634d8b58b5a/console-operator/3.log"
Nov 03 21:39:20.938824 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:20.938792    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shbqc_88128ae2-7d2c-41a4-a0d1-ceb286a7d80f/ovn-acl-logging/0.log"
Nov 03 21:39:21.090719 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.090686    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"]
Nov 03 21:39:21.091260 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.091245    5196 cpu_manager.go:414] "containerMap: removing container" podUID="78bd51b9-267e-48d0-9f6d-8831e5201fb2" containerName="revoke-keys"
Nov 03 21:39:21.091323 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.091261    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78bd51b9-267e-48d0-9f6d-8831e5201fb2" containerName="revoke-keys"
Nov 03 21:39:21.091382 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.091372    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="78bd51b9-267e-48d0-9f6d-8831e5201fb2" containerName="revoke-keys"
Nov 03 21:39:21.094761 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.094741    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.145387 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.145352    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:39:21.148599 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.148516    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"]
Nov 03 21:39:21.159060 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.159020    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.159254 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.159077    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.159393 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.159272    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvrl\" (UniqueName: \"kubernetes.io/projected/591082c2-e58c-431c-91ba-edb93e4f03a0-kube-api-access-2dvrl\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.175018 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.174989    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tssvb\""
Nov 03 21:39:21.259869 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.259833    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.260049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.259886    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.260049 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.259942    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvrl\" (UniqueName: \"kubernetes.io/projected/591082c2-e58c-431c-91ba-edb93e4f03a0-kube-api-access-2dvrl\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.260552 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.260519    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.262729 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.262700    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.313955 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.313920    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvrl\" (UniqueName: \"kubernetes.io/projected/591082c2-e58c-431c-91ba-edb93e4f03a0-kube-api-access-2dvrl\") pod \"maas-api-revoke-sub-6b19338f-nx597\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.408271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.408184    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:21.789081 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:21.788992    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"]
Nov 03 21:39:21.796787 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:39:21.796759    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591082c2_e58c_431c_91ba_edb93e4f03a0.slice/crio-e5a419bdf2e2ea27ecdf2cf7de59fd6e53b12722a50c5f8e83f9f8705b1014ae WatchSource:0}: Error finding container e5a419bdf2e2ea27ecdf2cf7de59fd6e53b12722a50c5f8e83f9f8705b1014ae: Status 404 returned error can't find the container with id e5a419bdf2e2ea27ecdf2cf7de59fd6e53b12722a50c5f8e83f9f8705b1014ae
Nov 03 21:39:22.618764 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:22.618722    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597" event={"ID":"591082c2-e58c-431c-91ba-edb93e4f03a0","Type":"ContainerStarted","Data":"4e1f07f1c33fd25e1077468d09b9069d444ce12f8ebdc94c053eab21457a0695"}
Nov 03 21:39:22.619481 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:22.619451    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597" event={"ID":"591082c2-e58c-431c-91ba-edb93e4f03a0","Type":"ContainerStarted","Data":"e5a419bdf2e2ea27ecdf2cf7de59fd6e53b12722a50c5f8e83f9f8705b1014ae"}
Nov 03 21:39:22.728046 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:22.727980    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597" podStartSLOduration=1.727965036 podStartE2EDuration="1.727965036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:22.725306704 +0000 UTC m=+4202.729265428" watchObservedRunningTime="2026-11-03 21:39:22.727965036 +0000 UTC m=+4202.731923760"
Nov 03 21:39:23.626501 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:23.626470    5196 generic.go:356] "Generic (PLEG): container finished" podID="591082c2-e58c-431c-91ba-edb93e4f03a0" containerID="4e1f07f1c33fd25e1077468d09b9069d444ce12f8ebdc94c053eab21457a0695" exitCode=0
Nov 03 21:39:23.626907 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:23.626546    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597" event={"ID":"591082c2-e58c-431c-91ba-edb93e4f03a0","Type":"ContainerDied","Data":"4e1f07f1c33fd25e1077468d09b9069d444ce12f8ebdc94c053eab21457a0695"}
Nov 03 21:39:24.797603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:24.797573    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:24.900972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:24.900884    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-openshift-ca\") pod \"591082c2-e58c-431c-91ba-edb93e4f03a0\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") "
Nov 03 21:39:24.901165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:24.901021    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-xks-ca\") pod \"591082c2-e58c-431c-91ba-edb93e4f03a0\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") "
Nov 03 21:39:24.901165 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:24.901118    5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591082c2-e58c-431c-91ba-edb93e4f03a0-kube-api-access-2dvrl\" (UniqueName: \"kubernetes.io/projected/591082c2-e58c-431c-91ba-edb93e4f03a0-kube-api-access-2dvrl\") pod \"591082c2-e58c-431c-91ba-edb93e4f03a0\" (UID: \"591082c2-e58c-431c-91ba-edb93e4f03a0\") "
Nov 03 21:39:24.901320 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:24.901300    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-openshift-ca" pod "591082c2-e58c-431c-91ba-edb93e4f03a0" (UID: "591082c2-e58c-431c-91ba-edb93e4f03a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:24.901507 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:24.901484    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-openshift-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:24.903379 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:24.903355    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-xks-ca" pod "591082c2-e58c-431c-91ba-edb93e4f03a0" (UID: "591082c2-e58c-431c-91ba-edb93e4f03a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:24.903543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:24.903520    5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591082c2-e58c-431c-91ba-edb93e4f03a0-kube-api-access-2dvrl" pod "591082c2-e58c-431c-91ba-edb93e4f03a0" (UID: "591082c2-e58c-431c-91ba-edb93e4f03a0"). InnerVolumeSpecName "kube-api-access-2dvrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:25.002579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:25.002541    5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591082c2-e58c-431c-91ba-edb93e4f03a0-maas-api-xks-ca\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:25.002579 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:25.002575    5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dvrl\" (UniqueName: \"kubernetes.io/projected/591082c2-e58c-431c-91ba-edb93e4f03a0-kube-api-access-2dvrl\") on node \"ip-10-0-128-49.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:25.639915 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:25.639883    5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597"
Nov 03 21:39:25.640138 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:25.639916    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nx597" event={"ID":"591082c2-e58c-431c-91ba-edb93e4f03a0","Type":"ContainerDied","Data":"e5a419bdf2e2ea27ecdf2cf7de59fd6e53b12722a50c5f8e83f9f8705b1014ae"}
Nov 03 21:39:25.640138 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:25.639998    5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a419bdf2e2ea27ecdf2cf7de59fd6e53b12722a50c5f8e83f9f8705b1014ae"
Nov 03 21:39:32.976348 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:32.976310    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7684488598-v9cnc_795cbb83-ddae-4f68-a84e-aad5d3a02340/authorino/0.log"
Nov 03 21:39:34.069195 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:34.069147    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h2grx"]
Nov 03 21:39:48.485185 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:39:48.485145    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:39:53.333822 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:53.333781    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-856cb7577-6mcxx_192e8de2-7248-4008-bf02-1873cad17c6a/manager/0.log"
Nov 03 21:39:53.671028 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:53.670954    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4jfx4_0e9476a2-3254-447f-bcfd-ffa8804e4d2f/manager/0.log"
Nov 03 21:39:59.443891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:39:59.443821    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:04.558961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:04.558922    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8f44875fd-kwb2n_67ee1ede-2dda-4079-80e3-e14ecf2790b5/manager/0.log"
Nov 03 21:40:05.442024 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:05.441985    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lkc7w_9cd606fe-f716-464a-bbf3-6e3eec674b5b/manager/0.log"
Nov 03 21:40:05.709181 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:05.709084    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7vlp8_b1a4914f-99fa-492e-9e2a-716fe58ab628/proxy/0.log"
Nov 03 21:40:06.327961 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:06.327926    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:06.935039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:06.935005    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-t7c66_78bd51b9-267e-48d0-9f6d-8831e5201fb2/revoke-keys/0.log"
Nov 03 21:40:07.139712 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:07.139679    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nx597_591082c2-e58c-431c-91ba-edb93e4f03a0/revoke-keys/0.log"
Nov 03 21:40:07.349811 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:07.349786    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-zjp9m_60ff5464-5661-479f-a4fb-3e0368f0979f/revoke-keys/0.log"
Nov 03 21:40:07.562378 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:07.562345    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-frjx5_a269e6a3-f734-4a59-a71d-764f59259a36/postgres/0.log"
Nov 03 21:40:08.927447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:08.927411    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7684488598-v9cnc_795cbb83-ddae-4f68-a84e-aad5d3a02340/authorino/0.log"
Nov 03 21:40:09.365942 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:09.365905    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zxpzr_b4244452-ce11-46a6-8909-ecbdab87137d/manager/0.log"
Nov 03 21:40:09.603895 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:09.603860    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vqvkc_c0a56f9e-a976-4628-8573-2a10b958deea/manager/0.log"
Nov 03 21:40:10.050881 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:10.050844    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mc5f5_6c78e9d1-87dc-417f-9b2e-87605826bb0a/manager/0.log"
Nov 03 21:40:10.390781 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:10.390748    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h2grx_9830db65-bd33-400b-805c-d8b27fbaf442/limitador/0.log"
Nov 03 21:40:10.605761 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:10.605729    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cntd4_7ae07717-de88-4d8f-b9f3-42b58dbf9bba/manager/0.log"
Nov 03 21:40:11.376047 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:11.376011    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-55fd485789zcp6f_7b04a8da-2555-4dfd-b88b-f86a86df7114/istio-proxy/0.log"
Nov 03 21:40:12.448899 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:12.448860    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:12.705776 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:12.705698    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-fm4zc_1cba3ff6-170a-45dc-8cf7-1517813237c9/istio-proxy/0.log"
Nov 03 21:40:13.388150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:13.388112    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-rrffz_e67311d6-9cf7-4985-abb1-1cc9763e6156/istio-proxy/1.log"
Nov 03 21:40:13.831915 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:13.831884    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-tcqsm_3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc/istio-proxy/0.log"
Nov 03 21:40:14.503704 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:14.503659    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v_84704e32-8624-49c8-b585-a12ee07f5496/istio-proxy/0.log"
Nov 03 21:40:18.485564 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:40:18.485520    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:40:24.714371 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:24.714328    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7pv4c_b39296fa-08ee-4504-b9db-ee97ab3e7d4b/payload-pre-processing/0.log"
Nov 03 21:40:24.932402 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:24.932367    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b785fc555-gj4ww_d289e2d2-7ef4-4218-80db-79ef4ea15d0e/payload-processing/0.log"
Nov 03 21:40:25.149885 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:25.149854    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7454997c-ccrh5_ee248b83-7c43-46fe-809c-69b61ca6254a/router/0.log"
Nov 03 21:40:25.686591 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:25.686553    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-rrffz_e67311d6-9cf7-4985-abb1-1cc9763e6156/istio-proxy/0.log"
Nov 03 21:40:26.510231 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:26.510205    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7nr6l_822db250-6afd-4838-b96d-b64f1b7dbfc9/main/0.log"
Nov 03 21:40:26.768472 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:26.768388    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-k4dtv_e24913d4-e48b-489c-ae78-1a091424e4fb/main/0.log"
Nov 03 21:40:27.279524 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:27.279493    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t_42330a88-2cf2-4b6e-a877-34cbcba782d6/main/0.log"
Nov 03 21:40:27.789868 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:27.789839    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m_178b98c9-47eb-4858-a069-469155095085/main/0.log"
Nov 03 21:40:28.011813 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:28.011783    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz_ed185af4-9540-4841-a9c3-bd14cf61e7e2/main/0.log"
Nov 03 21:40:28.888983 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:28.888948    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7684488598-v9cnc_795cbb83-ddae-4f68-a84e-aad5d3a02340/authorino/0.log"
Nov 03 21:40:33.959112 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:33.959054    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:34.106226 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:34.106189    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:37.493970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:37.493929    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-856cb7577-6mcxx_192e8de2-7248-4008-bf02-1873cad17c6a/manager/0.log"
Nov 03 21:40:37.840428 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:37.840385    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4jfx4_0e9476a2-3254-447f-bcfd-ffa8804e4d2f/manager/0.log"
Nov 03 21:40:38.679708 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:38.679666    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8f44875fd-kwb2n_67ee1ede-2dda-4079-80e3-e14ecf2790b5/manager/0.log"
Nov 03 21:40:39.129164 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:39.129120    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lkc7w_9cd606fe-f716-464a-bbf3-6e3eec674b5b/manager/0.log"
Nov 03 21:40:39.376970 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:39.376934    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7vlp8_b1a4914f-99fa-492e-9e2a-716fe58ab628/proxy/0.log"
Nov 03 21:40:40.439068 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:40.439034    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-t7c66_78bd51b9-267e-48d0-9f6d-8831e5201fb2/revoke-keys/0.log"
Nov 03 21:40:40.653409 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:40.653378    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nx597_591082c2-e58c-431c-91ba-edb93e4f03a0/revoke-keys/0.log"
Nov 03 21:40:40.864118 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:40.864072    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-zjp9m_60ff5464-5661-479f-a4fb-3e0368f0979f/revoke-keys/0.log"
Nov 03 21:40:41.066104 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:41.066058    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-frjx5_a269e6a3-f734-4a59-a71d-764f59259a36/postgres/0.log"
Nov 03 21:40:42.119273 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:42.119232    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7684488598-v9cnc_795cbb83-ddae-4f68-a84e-aad5d3a02340/authorino/0.log"
Nov 03 21:40:42.519847 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:42.519769    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zxpzr_b4244452-ce11-46a6-8909-ecbdab87137d/manager/0.log"
Nov 03 21:40:42.735400 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:42.735363    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vqvkc_c0a56f9e-a976-4628-8573-2a10b958deea/manager/0.log"
Nov 03 21:40:43.162555 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:43.162520    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mc5f5_6c78e9d1-87dc-417f-9b2e-87605826bb0a/manager/0.log"
Nov 03 21:40:43.410972 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:43.410945    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h2grx_9830db65-bd33-400b-805c-d8b27fbaf442/limitador/0.log"
Nov 03 21:40:43.623860 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:43.623819    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cntd4_7ae07717-de88-4d8f-b9f3-42b58dbf9bba/manager/0.log"
Nov 03 21:40:44.179891 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:44.179862    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-55fd485789zcp6f_7b04a8da-2555-4dfd-b88b-f86a86df7114/istio-proxy/0.log"
Nov 03 21:40:45.471074 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:45.471044    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-fm4zc_1cba3ff6-170a-45dc-8cf7-1517813237c9/istio-proxy/0.log"
Nov 03 21:40:46.137331 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:46.137297    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-rrffz_e67311d6-9cf7-4985-abb1-1cc9763e6156/istio-proxy/1.log"
Nov 03 21:40:46.587135 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:46.587107    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-tcqsm_3c1e7367-a2c0-4190-9b38-20d1dcfb0ddc/istio-proxy/0.log"
Nov 03 21:40:47.259778 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:47.259740    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-648cc4b8c5-2jg9v_84704e32-8624-49c8-b585-a12ee07f5496/istio-proxy/0.log"
Nov 03 21:40:47.472681 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:47.472646    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7pv4c_b39296fa-08ee-4504-b9db-ee97ab3e7d4b/payload-pre-processing/0.log"
Nov 03 21:40:47.697533 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:47.697499    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b785fc555-gj4ww_d289e2d2-7ef4-4218-80db-79ef4ea15d0e/payload-processing/0.log"
Nov 03 21:40:47.906503 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:47.906476    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7454997c-ccrh5_ee248b83-7c43-46fe-809c-69b61ca6254a/router/0.log"
Nov 03 21:40:48.484602 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:40:48.484557    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:40:48.678686 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:48.678656    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7nr6l_822db250-6afd-4838-b96d-b64f1b7dbfc9/main/0.log"
Nov 03 21:40:48.952661 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:48.952633    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-k4dtv_e24913d4-e48b-489c-ae78-1a091424e4fb/main/0.log"
Nov 03 21:40:49.382447 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:49.382414    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tp62t_42330a88-2cf2-4b6e-a877-34cbcba782d6/main/0.log"
Nov 03 21:40:49.853271 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:49.853236    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tlp8m_178b98c9-47eb-4858-a069-469155095085/main/0.log"
Nov 03 21:40:50.107669 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:50.107579    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-z4swz_ed185af4-9540-4841-a9c3-bd14cf61e7e2/main/0.log"
Nov 03 21:40:59.725206 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:40:59.725176    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:18.484874 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:41:18.484833    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:41:22.883765 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:41:22.883723    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:26.309151 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:41:26.309054    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8f44875fd-kwb2n_67ee1ede-2dda-4079-80e3-e14ecf2790b5/manager/0.log"
Nov 03 21:41:26.861706 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:41:26.861674    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8f44875fd-kwb2n_67ee1ede-2dda-4079-80e3-e14ecf2790b5/manager/0.log"
Nov 03 21:41:27.531240 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:41:27.531211    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b785fc555-gj4ww_d289e2d2-7ef4-4218-80db-79ef4ea15d0e/payload-processing/0.log"
Nov 03 21:41:27.743232 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:41:27.743201    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b785fc555-gj4ww_d289e2d2-7ef4-4218-80db-79ef4ea15d0e/payload-processing/0.log"
Nov 03 21:41:48.485449 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:41:48.485401    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:42:00.075392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:00.075353    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:00.332197 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:00.332080    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:06.136523 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:06.136485    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:06.285744 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:06.285697    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:18.485821 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:42:18.485778    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:42:21.562256 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:21.562218    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8f44875fd-kwb2n_67ee1ede-2dda-4079-80e3-e14ecf2790b5/manager/0.log"
Nov 03 21:42:22.117902 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:22.117866    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8f44875fd-kwb2n_67ee1ede-2dda-4079-80e3-e14ecf2790b5/manager/0.log"
Nov 03 21:42:22.818470 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:22.818443    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b785fc555-gj4ww_d289e2d2-7ef4-4218-80db-79ef4ea15d0e/payload-processing/0.log"
Nov 03 21:42:23.032869 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:23.032841    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b785fc555-gj4ww_d289e2d2-7ef4-4218-80db-79ef4ea15d0e/payload-processing/0.log"
Nov 03 21:42:23.382655 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:23.382619    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:23.559950 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:23.559903    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:28.914405 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:28.914373    5196 scope.go:122] "RemoveContainer" containerID="db740c5d0e1ff3d5122c2d4dc2a09abfbfb555855a0ba9560671b265cb3054ff"
Nov 03 21:42:33.021335 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:33.021301    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8w5qd_428ef791-c562-4786-b095-7c3b1113bfc8/global-pull-secret-syncer/0.log"
Nov 03 21:42:34.148290 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:34.148259    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6vt8w_1d9f21de-9ec8-48b7-aa66-5945a79d95e4/connection-checker/0.log"
Nov 03 21:42:34.556149 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:34.556038    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hjc54_68c418d8-0eb1-44a4-9c0f-59d3d7d724af/connection-checker/0.log"
Nov 03 21:42:34.965967 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:34.965941    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qpwf2_a533118b-20c1-478a-b7e7-42dd7b4f200c/connection-checker/0.log"
Nov 03 21:42:35.423168 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:35.423133    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log"
Nov 03 21:42:36.774560 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:36.774528    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-49.us-west-2.compute.internal_5b0fa7d4d6c53e4753bcd43d896ecfe1/haproxy/0.log"
Nov 03 21:42:41.597336 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:41.597299    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7684488598-v9cnc_795cbb83-ddae-4f68-a84e-aad5d3a02340/authorino/0.log"
Nov 03 21:42:41.909794 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:41.909705    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zxpzr_b4244452-ce11-46a6-8909-ecbdab87137d/manager/0.log"
Nov 03 21:42:41.937131 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:41.937083    5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"]
Nov 03 21:42:41.937680 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:41.937661    5196 cpu_manager.go:414] "containerMap: removing container" podUID="591082c2-e58c-431c-91ba-edb93e4f03a0" containerName="revoke-keys"
Nov 03 21:42:41.937680 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:41.937677    5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591082c2-e58c-431c-91ba-edb93e4f03a0" containerName="revoke-keys"
Nov 03 21:42:41.937835 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:41.937762    5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="591082c2-e58c-431c-91ba-edb93e4f03a0" containerName="revoke-keys"
Nov 03 21:42:41.941170 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:41.941151    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:41.992883 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:41.992850    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"]
Nov 03 21:42:42.004228 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.004199    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-sys\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.004392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.004263    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gz4\" (UniqueName: \"kubernetes.io/projected/7185f2a8-30ca-4f99-9347-b95cfac5a006-kube-api-access-f4gz4\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.004392 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.004363    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-lib-modules\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.004516 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.004493    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-podres\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.004567 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.004553    5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-proc\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.036072 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.036031    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dd722\"/\"openshift-service-ca.crt\""
Nov 03 21:42:42.036272 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.036080    5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dd722\"/\"default-dockercfg-85g9m\""
Nov 03 21:42:42.036272 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.036172    5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dd722\"/\"kube-root-ca.crt\""
Nov 03 21:42:42.106157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106118    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-podres\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.106157 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106164    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-proc\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.106413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106193    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-sys\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.106413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106253    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gz4\" (UniqueName: \"kubernetes.io/projected/7185f2a8-30ca-4f99-9347-b95cfac5a006-kube-api-access-f4gz4\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.106413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106261    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-proc\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.106413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106260    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-podres\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.106413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106299    5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-lib-modules\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.106413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106362    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-lib-modules\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.106413 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.106302    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7185f2a8-30ca-4f99-9347-b95cfac5a006-sys\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.160441 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.160360    5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gz4\" (UniqueName: \"kubernetes.io/projected/7185f2a8-30ca-4f99-9347-b95cfac5a006-kube-api-access-f4gz4\") pod \"perf-node-gather-daemonset-p76jz\" (UID: \"7185f2a8-30ca-4f99-9347-b95cfac5a006\") " pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.254055 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.254017    5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:42.314222 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.314071    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vqvkc_c0a56f9e-a976-4628-8573-2a10b958deea/manager/0.log"
Nov 03 21:42:42.629543 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.629499    5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"]
Nov 03 21:42:42.636891 ip-10-0-128-49 kubenswrapper[5196]: W1103 21:42:42.636858    5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7185f2a8_30ca_4f99_9347_b95cfac5a006.slice/crio-2493683b12e5dbb5dfcae40c252027c7f51649f844aeb3af1cb43f9fe042be65 WatchSource:0}: Error finding container 2493683b12e5dbb5dfcae40c252027c7f51649f844aeb3af1cb43f9fe042be65: Status 404 returned error can't find the container with id 2493683b12e5dbb5dfcae40c252027c7f51649f844aeb3af1cb43f9fe042be65
Nov 03 21:42:42.941841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.941741    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz" event={"ID":"7185f2a8-30ca-4f99-9347-b95cfac5a006","Type":"ContainerStarted","Data":"c90b8be329fa473f8090d84bd7a4d0014fbd07f3c3af86f66f6e7c0de8186fd3"}
Nov 03 21:42:42.941841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.941796    5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz" event={"ID":"7185f2a8-30ca-4f99-9347-b95cfac5a006","Type":"ContainerStarted","Data":"2493683b12e5dbb5dfcae40c252027c7f51649f844aeb3af1cb43f9fe042be65"}
Nov 03 21:42:42.941841 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:42.941821    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:43.049878 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:43.049819    5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz" podStartSLOduration=2.049806935 podStartE2EDuration="2.049806935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:43.048064996 +0000 UTC m=+4403.052023717" watchObservedRunningTime="2026-11-03 21:42:43.049806935 +0000 UTC m=+4403.053765659"
Nov 03 21:42:43.171286 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:43.171252    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mc5f5_6c78e9d1-87dc-417f-9b2e-87605826bb0a/manager/0.log"
Nov 03 21:42:43.532039 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:43.532012    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h2grx_9830db65-bd33-400b-805c-d8b27fbaf442/limitador/0.log"
Nov 03 21:42:44.013150 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:44.013117    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cntd4_7ae07717-de88-4d8f-b9f3-42b58dbf9bba/manager/0.log"
Nov 03 21:42:48.485551 ip-10-0-128-49 kubenswrapper[5196]: E1103 21:42:48.485507    5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc4mt" containerName="extractor"
Nov 03 21:42:48.961079 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:48.961048    5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dd722/perf-node-gather-daemonset-p76jz"
Nov 03 21:42:49.384603 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:49.384568    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4g2kg_a3b4704b-2627-4530-8100-838f2d8b7d31/cluster-monitoring-operator/0.log"
Nov 03 21:42:51.069861 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:51.069830    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6947bdbb66-klklq_e697a4b7-ab69-445d-9354-4e8ad77ef608/metrics-server/0.log"
Nov 03 21:42:51.476160 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:51.476075    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-mlgwh_bf1dbf60-0a31-48f3-aa2b-6d32468e6054/monitoring-plugin/0.log"
Nov 03 21:42:51.889556 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:51.889530    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcl5k_2f61a647-3229-4c6e-b181-a44abbea314e/node-exporter/0.log"
Nov 03 21:42:52.291123 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:52.291025    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcl5k_2f61a647-3229-4c6e-b181-a44abbea314e/kube-rbac-proxy/0.log"
Nov 03 21:42:52.693568 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:42:52.693542    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcl5k_2f61a647-3229-4c6e-b181-a44abbea314e/init-textfile/0.log"
Nov 03 21:43:00.379433 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:43:00.379340    5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmhst_e3d788c1-23d6-4815-8e9f-8d4aebfe1bd3/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:02.159758 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:43:02.159726    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf8575ffd-g9cmq_06e732ae-04e9-4ae2-83da-3664f681018c/thanos-query/0.log"
Nov 03 21:43:02.562984 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:43:02.562906    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf8575ffd-g9cmq_06e732ae-04e9-4ae2-83da-3664f681018c/kube-rbac-proxy-web/0.log"
Nov 03 21:43:02.967908 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:43:02.967872    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf8575ffd-g9cmq_06e732ae-04e9-4ae2-83da-3664f681018c/kube-rbac-proxy/0.log"
Nov 03 21:43:03.370644 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:43:03.370607    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf8575ffd-g9cmq_06e732ae-04e9-4ae2-83da-3664f681018c/prom-label-proxy/0.log"
Nov 03 21:43:03.797687 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:43:03.797603    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf8575ffd-g9cmq_06e732ae-04e9-4ae2-83da-3664f681018c/kube-rbac-proxy-rules/0.log"
Nov 03 21:43:04.201890 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:43:04.201848    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cf8575ffd-g9cmq_06e732ae-04e9-4ae2-83da-3664f681018c/kube-rbac-proxy-metrics/0.log"
Nov 03 21:43:04.226272 ip-10-0-128-49 kubenswrapper[5196]: I1103 21:43:04.226232    5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-55fd485789zcp6f_7b04a8da-2555-4dfd-b88b-f86a86df7114/istio-proxy/0.log"